The 100-Prompt AI Search Experiment: Reproducible Protocol

A 100-prompt AI search experiment can measure brand discovery, citations, recommendations and volatility across platforms—if the prompts are fixed before collection and repeated under documented conditions. This article publishes the complete TotalAuthority protocol. It does not claim that a proprietary 100-prompt collection has already happened.

What the Experiment Should Reveal

The study should answer:

• How consistently do brands appear across repeated runs?
• Which source domains and pages recur?
• How often are brands recommended without an owned citation?
• How much do results differ by platform and buyer stage?
• Which answers contain material brand errors?
• How large is the normal run-to-run variance?

The purpose is not to crown a universal winner. It is to characterize observed distributions for a defined market and date range.

Design the 100 Prompts

Use five balanced modules.

Module A: Problem education — 20 prompts

Questions about causes, risks, terminology and decision criteria. These establish which publishers and experts are used for educational answers.

Module B: Solution exploration — 20 prompts

Questions comparing methods, service types or implementation approaches without naming providers.

Module C: Provider discovery — 20 prompts

Natural requests for providers that meet explicit audience, location, budget, specialization or governance requirements.

Module D: Comparison and validation — 20 prompts

Questions that compare alternatives, test suitability, ask about limitations or seek independent proof.

Module E: Branded accuracy — 20 prompts

Questions about products, people, locations, credentials, policies and positioning. Keep this module separate because the brand name is prompted.

Generate Prompts Responsibly

Source questions from sales calls, support logs, keyword research, industry experts and real buyer journeys. Remove confidential information. Have subject-matter experts review wording for plausibility.

Freeze version 1.0 before collection. If a wording defect is discovered, document the revision and restart affected matched observations rather than silently editing the prompt.

Choose the Platforms

Select consumer surfaces based on audience use. Record product, mode and model label where visible. A robust design could include ChatGPT search, Perplexity, Gemini and one Microsoft or Google Search AI surface, but only if the team can collect them consistently.

Do not treat “Google AI Overview” and “Gemini” as interchangeable. Do not use an API model without retrieval as a proxy for a consumer search product.

Decide the Repetitions

The minimum should be driven by desired precision and observed variance. Five independent runs per prompt per platform is a practical starting point for an exploratory study.

At four platforms, the design produces:

100 prompts × 4 platforms × 5 runs = 2,000 answer observations

Run order should be randomized to reduce time-of-day and platform-event effects. Reset conversation context between independent observations.

Capture the Environment

For every run, store:

• prompt ID and exact text;
• platform and surface;
• collection time and timezone;
• country and language;
• account state;
• conversation state;
• search or browsing indicator where visible;
• raw answer;
• visible sources and URLs;
• errors, refusals and invalid-run reason.

Screenshots help audit presentation, while machine-readable answer and URL fields support analysis.

Define the Outcomes

Mention

The brand appears by an approved official name, product name or unambiguous alias.

Citation

A visible source link points to the owned domain or an explicitly defined earned source.

Recommendation

The answer positively includes the brand as a suitable option for the requested need. A mere list or negative warning is not automatically a recommendation.

Accuracy

Material branded claims are correct according to the dated fact ledger. Classify minor, material and critical errors.

Sentiment and fit

Record positive, neutral, mixed or negative context, plus whether the recommendation is appropriate for the audience.

The AI brand sentiment guide supplies the full taxonomy.

Classify the Sources

Resolve redirect and canonical variants. Then classify each source as owned, editorial, government, academic, association, review, community, retailer, reference or other.

Preserve both domain and page-level data. A recurring publisher domain may contribute different articles across platforms.

Quality Assurance

Train reviewers on a labeled example set. Double-review at least 10% of observations and all critical errors. Report agreement and adjudication rules.

Automated classifiers can assist with scale, but save their instructions, model versions and confidence. Do not let the same model grade itself without human checks on high-risk outcomes.

Analysis Plan

Calculate:

• mention, citation and recommendation rates with intervals;
• accuracy rate conditional on a brand mention;
• prompted and unprompted presence separately;
• platform and module comparisons;
• run-to-run consistency;
• source-category share;
• domain and page overlap;
• competitor share of voice;
• critical misinformation incidence.

Use medians and distributions where brand-level results are skewed. Apply multiple-comparison controls if testing many hypotheses.

Volatility Measures

For each prompt-platform pair, calculate the proportion of repeated runs in which the outcome changes. A prompt mentioned in two of five runs has a 40% observed mention rate, not a fixed rank.

Track source-set Jaccard overlap across runs and the standard deviation or interval around key rates. The AI visibility measurement frequency guide explains how the experiment becomes ongoing monitoring.

Result Tables to Publish

No values appear until the data collection is complete.

Release Package

The finished experiment should publish:

• prompt list or documented template;
• collection dates and surface definitions;
• data dictionary;
• privacy-safe observation dataset;
• canonicalized citation table;
• analysis notebook or code;
• human-review rubric;
• exclusions and failures;
• limitations and version history.

Common Failure Modes

• Running each prompt once.
• Mixing branded and unbranded prompts.
• Changing prompts mid-study.
• Combining incompatible product modes.
• Treating source order as ranking.
• Counting inaccurate mentions as success.
• Ignoring invalid runs.
• Publishing only positive examples.
• Inferring permanent ranking factors from one collection window.

Frequently Asked Questions

Why exactly 100 prompts?

It is a manageable, communicable design—not a statistical law. A smaller focused set can be better than 100 weak prompts.

Can the same study compare industries?

Only with industry-specific modules, sufficient samples and careful normalization. Do not use identical wording where buyer decisions differ.

Should the prompts include location?

Yes for local or regulated services, but treat each market as a separate segment and document how location was set.

When will the TotalAuthority results be available?

After the fixed prompts, repeated runs, human QA and reproducible release package are complete. Until then, this page is the public protocol.

Sources

• Don't Measure Once: Measuring Visibility in AI Search
• Quantifying Uncertainty in AI Visibility
• From Citation Selection to Citation Absorption
• NIST Generative AI Profile