AI Claims Lab: a fixed protocol for testing PIM vendor AI claims

What is the AI Claims Lab and what is its status?

The AI Claims Lab is the PIMvendors protocol for testing PIM vendor AI claims on a buyer's own data. Every vendor processes the same 200 products, under the same rules, in the same time window. The results are measured per attribute and per task.

Status: the first published vendor round is planned. No results are published yet. This page publishes the protocol and the scoring sheet so buyers can use them in their own selections now.

The lab exists because AI demos predict little. Demo data is clean and chosen to succeed. The AI in PIM pillar rates which AI use cases are production-ready; this protocol checks whether a specific vendor delivers them on a specific catalogue.

The protocol tests the four use cases PIMvendors rates as production-ready with review:

1. Attribute extraction from unstructured sources.
2. Classification into an internal taxonomy and one trade standard.
3. Translation into two target languages.
4. Channel-specific descriptions for two channels.

It also checks two boundaries: whether a vendor leaves fields empty when the source is silent, and whether it stays away from compliance attributes. Both connect to the rule from AI attribute extraction: no source, no value. The classification and translation tasks follow the metrics set out in AI auto-classification and AI translation.

How is the 200-product test set built?

The buyer provides 200 products, stratified so that every known AI failure mode appears several times. The buyer's stewards build an answer key that vendors never see.

For each product the buyer supplies the source documents and the target attribute list for its product family. The approved values go into a held-back answer key. Translation and description tasks use approved source text and attributes as input, with native reviewers ready to post-edit.

Keep the set fixed for the whole round. Changing products between vendors breaks comparability.

Which tasks and metrics does the protocol use?

Each task has its own metrics, and every task records time per item and cost per item. Averages across attributes or languages are reported only next to the detailed figures.

Definitions used in scoring:

• Precision: correct filled values divided by all filled values, per attribute.
• Recall: correct filled values divided by values present in the source, per attribute.
• Human edit rate: share of characters changed by the reviewer, using an edit-distance measure.
• Unsupported claim: any statement not backed by a supplied attribute.

What are the test rules?

Every vendor gets identical inputs, the same window and no access to the answer key. Vendors configure their product as they would for a customer and declare everything they used.

1. Inputs are delivered to all vendors at the same time, under a data processing agreement.
2. The test window is fixed, for example 10 working days.
3. Vendors may configure prompts, glossaries, attribute targets and thresholds.
4. Vendors may not edit outputs by hand.
5. Vendors declare the model, configuration, hosting region and credits consumed, including regenerations.
6. Outputs are returned in the buyer's template and scored by the buyer's reviewers, blind to vendor name where possible.
7. Reviewers use the same instructions and post-editing standard for every vendor.

A vendor that declines to run the test, or asks to use its own sample data, should be recorded as such in the scoring sheet. The demo scorecard can carry the AI results into the overall evaluation.

Scoring sheet

Copy the sheet below and fill one column per vendor. Keep per-attribute detail in a second tab; this summary carries the figures the buying committee needs.

Per-attribute tab columns: attribute, stratum, values in source, values filled, values correct, precision, recall.

Read the sheet from the bottom up. Cost per item and reviewer time show what the AI costs to run; the task rows show what it delivers for that cost. A vendor with slightly lower precision and half the review time can be the better choice. Discuss trade-offs like that in the buying committee before anyone sees vendor names.

How will PIMvendors run public vendor rounds?

PIMvendors plans to run public rounds on a shared dataset, invite vendors on equal terms and give each vendor a right of reply before publication. The first round is planned; no results are published yet.

The planned process:

1. PIMvendors prepares a 200-product dataset using the same strata, as an anonymised composite. No customer names appear.
2. Vendors in the relevant segment are invited on identical terms. Participation carries no fee and no sponsorship link.
3. Vendors that decline are listed as declined.
4. Results are scored by PIMvendors reviewers under the rules above.
5. Each vendor receives its results and scoring notes before publication and can correct factual errors.
6. Each vendor can submit a written response, published next to its results.

The rounds follow the hub's neutrality policy. Results will appear per task and per vendor, with no overall star ratings or scores out of 10.