How to track whether ChatGPT and Claude mention your company, starting with Claude

Step 1. Say who you are and what buyers ask

One block at the top of the script holds everything you change. Each company has a list of spellings, because an assistant writes “Plausible” where a brand page says “Plausible Analytics”: in the saved answers, Claude wrote the short form “Plausible” in all six. A spelling matches as a whole word and is case-sensitive, so the everyday word “plausible” does not count, and neither does “Matomoo”.

# ---- EDIT THIS BLOCK TO RUN IT ON YOUR OWN COMPANY ------------------------------------------
# "Matomo" and the four tools below are a public stand-in for your company and your competitors.
# Matomo is not a client and has no part in this guide.
COMPANY = "Matomo"
# Every spelling that counts as your company. Matched whole-word and case-sensitive, so "Plausible"
# counts but the ordinary word "plausible" does not. Assistants shorten names: add the short form.
COMPANY_NAMES = ["Matomo", "matomo.org"]
# Competitors: display name -> the spellings that count as that competitor.
COMPETITORS = {
    "Plausible Analytics": ["Plausible Analytics", "Plausible", "plausible.io"],
    "Fathom Analytics": ["Fathom Analytics", "Fathom", "usefathom.com"],
    "Simple Analytics": ["Simple Analytics", "simpleanalytics.com"],
    "Google Analytics": ["Google Analytics", "GA4"],
}
# The questions a buyer would type. Do not put your own company name in them. At most 10.
QUESTIONS = [
    "What are the best privacy-friendly website analytics tools?",
    "Which website analytics tool works without a cookie consent banner?",
    "What is a good self-hosted web analytics platform?",
    "Which web analytics tools are GDPR friendly for a European website?",
    "What can I use to replace Google Analytics on a small business website?",
    "Which open source web analytics tool should a nonprofit use?",
]
# ---------------------------------------------------------------------------------------------

The script refuses to start if a question contains one of your spellings, because a question that names you leads the answer to you. It also refuses more than ten questions.

Step 2. Let Claude answer like an assistant

Each question goes to the Messages API with the web search tool (read 2026-11-03) and max_uses set to 3, so Claude searches at most three times per question. Claude must reply with JSON: its answer, and the list of companies it named. The model is claude-haiku-5-5, set in one constant, because answering and listing names does not need a larger model. Web search costs $10 per 1,000 searches plus token costs, and Haiku 5.5 costs $0.10 per million input tokens and $0.50 per million output tokens for prompts up to 100,000 tokens (Anthropic’s pricing page, read 2026-11-03). The script prints the tokens and the cost. A run is capped at ten questions and three searches each, and the script also refuses to start if its worst-case estimate passes $2.00. At the default limits that estimate is far below $2.00, so the guard only matters if you raise them.

def ask_claude(client, question):
    resp = client.messages.create(
        # no temperature: claude-haiku-5-5 answers 400 "temperature is deprecated for this model"
        model=MODEL, max_tokens=1500, system=SYSTEM,
        tools=[{"type": WEB_SEARCH_TOOL, "name": "web_search", "max_uses": MAX_SEARCHES_PER_QUESTION}],
        messages=[{"role": "user", "content": question}],
    )
    raw = "".join(b.text for b in resp.content if b.type == "text").strip()
    queries = [b.input.get("query", "") for b in resp.content if b.type == "server_tool_use"]
    searches = getattr(getattr(resp.usage, "server_tool_use", None), "web_search_requests", None) or len(queries)
    return {"question": question, "model": MODEL, "date": dt.date.today().isoformat(),
            "prompt": question, "system": SYSTEM, "raw_answer": raw, "searches": searches,
            "tokens_in": resp.usage.input_tokens, "tokens_out": resp.usage.output_tokens}

The script sets no temperature. When we tried temperature=0 on this model, the API answered with a 400 error that says temperature is deprecated for this model. So two live runs can word answers differently, which is why every answer is saved and why --replay exists.

Step 3. Recount every name with plain code

Claude’s own list is not trusted. The script looks for each tracked company in the answer text, and compares that with what Claude listed. A name Claude listed that the answer text does not contain is reported and not counted. A reply that is not the JSON the script expects is reported, never skipped.

def mentions(text, names):
    """Whole-word, case-sensitive match of any spelling in `names`. 'Plausibly' is not 'Plausible'."""
    return any(re.search(r"(?<![\w.])" + re.escape(n) + r"(?![\w])", text) for n in names)

def check(records):
    """The plain-code check. Returns per-question results and the company tallies."""
    results = []
    for i, r in enumerate(records, 1):
        answer, named, problem = parse(r["raw_answer"])
        res = {"n": i, "question": r["question"], "problem": problem, "flags": [], "found": [], "answer": answer}
        if not problem:
            for c in [COMPANY, *COMPETITORS]:
                if mentions(answer, aliases_of(c)):
                    res["found"].append(c)
            for listed in named:
                known = next((c for c in [COMPANY, *COMPETITORS] if listed == c or listed in aliases_of(c)), None)
                if known and known not in res["found"]:
                    res["flags"].append(f"Claude listed {listed!r} but the answer text never names it: not counted")
                elif not known and not re.search(r"(?<![\w.])" + re.escape(listed) + r"(?![\w])", answer):
                    res["flags"].append(f"Claude listed {listed!r} but the answer text never says it")
        results.append(res)
    return results

Run 2: plant three errors and see whether the check notices

Command: python3 ai_mention_check.py --replay ai_mention_check-answers.json --plant-error. It reads the answers saved by run 1, damages three of them as a careless model or a bad copy would, and runs the same checks. No key and no cost.

model claude-haiku-5-5, run date 2026-11-03, 6 questions
tokens in 98438, tokens out 7179, web searches 7, cost $0.0834
tracking: Matomo; competitors: Plausible Analytics, Fathom Analytics, Simple Analytics, Google Analytics

Q1: What are the best privacy-friendly website analytics tools?
    names: Plausible Analytics, Fathom Analytics, Simple Analytics, Google Analytics
    CAUGHT: Claude listed 'Matomo' but the answer text never names it: not counted
Q2: Which website analytics tool works without a cookie consent banner?
    names: Matomo, Plausible Analytics, Fathom Analytics, Simple Analytics
    CAUGHT: Claude listed 'Google Analytics' but the answer text never names it: not counted
Q3: What is a good self-hosted web analytics platform?
    names: Matomo, Plausible Analytics, Google Analytics
Q4: Which web analytics tools are GDPR friendly for a European website?
    REPLY PROBLEM: reply is not the expected JSON (JSONDecodeError)
Q5: What can I use to replace Google Analytics on a small business website?
    names: Matomo, Plausible Analytics, Fathom Analytics, Simple Analytics, Google Analytics
    CAUGHT: Claude listed 'Cloudflare Web Analytics' but the answer text never says it
Q6: Which open source web analytics tool should a nonprofit use?
    names: Matomo, Plausible Analytics, Google Analytics

company                questions naming it   share
Plausible Analytics                5 of 5    100.0%
Google Analytics                   4 of 5     80.0%
Matomo (yours)                     4 of 5     80.0%
Fathom Analytics                   3 of 5     60.0%
Simple Analytics                   3 of 5     60.0%

answers that parsed: 5 of 6; answers caught for a person to read: 4
RESULT: CHECK FAILED, read the caught answers before using these counts

Each planted error trips a different part of the check. The near-miss “Matomoo” and the added company both trip the comparison of Claude’s list with the answer text. The cut-off reply trips the JSON parse. The counts fall from six answers to five, because the answer that did not parse is left out and said so. The result line reads CHECK FAILED, which means a person reads the caught answers before using the counts.

Method

• Questions: the six in the script, none of which names Matomo. The same six every run.
• Model and tool: claude-haiku-5-5 with the web search tool (web_search_20250305), at most 3 searches per question, at most 10 questions a run. No temperature is set, because the model rejected it when we tried on 2026-11-03.
• What counts as a mention: a spelling from your list found as a whole word, with the same capitals, in Claude’s answer text. A mention does not say whether the answer is favorable, and a name in Claude’s list alone does not count.
• Mapping written out: Matomo: Matomo, matomo.org. Plausible Analytics: Plausible Analytics, Plausible, plausible.io. Fathom Analytics: Fathom Analytics, Fathom, usefathom.com. Simple Analytics: Simple Analytics, simpleanalytics.com. Google Analytics: Google Analytics, GA4. Change any of it in the script.
• Counts: questions whose answer named the company, over answers that parsed. Output is sorted by count, then name.
• Checks on the answers, each with the planted fault that trips it: a name Claude listed that its answer does not contain (the near-miss “Matomoo”, an added company); a reply that is not the expected JSON (a cut-off reply).
• Safeguards no run trips: the script stops before it asks if a question names your company or there are more than ten questions, and an internal assertion stops it if a count ever exceeds the answers checked.
• Cost: tokens and searches are read from the API response and priced at Anthropic’s price list (read 2026-11-03).

Limits

• This is not a client result. Matomo and the other four are public products used as a stand-in, and nothing here describes how any of them is doing. Data Meaning has no connection to them.
• Six questions are a small sample. The counts say what Claude did on these six questions on one day. They are not a ranking of the five tools.
• One assistant. Claude called through the API with web search is not the same as the Claude app, ChatGPT, Gemini or Copilot, and what people see there can differ.
• Answers vary. The same question can name different companies on another day or in another run. Web search results change over time.
• Matching is by name. The script does not read tone or accuracy. A short form such as “Plausible” at the start of a sentence as an ordinary adjective would be counted, so read the answers behind a surprising count.
• Counting does not change what assistants say. We can’t promise that any assistant will name your company. Nobody controls what AI assistants say.

Tested on

When this is not enough

• Six questions once is a snapshot. To see a trend you need the same questions every week, across more than one assistant, with a saved baseline. The In-House AI Marketing Engine’s AI search visibility part is “A weekly check of 20 buyer questions across five AI assistants, a baseline report, and a loop that writes guides.”
• Knowing who gets named does not make you get named. What assistants find about your company on the web is one thing that can change their answers. We can’t say how much, and we can’t promise any assistant will name you.
• A person still has to read the flags. The check catches a name that is not in the answer. It cannot tell you whether the answer is right.