Data samples: how to test a dataset before you buy

A data sample is the cheapest test you will ever run on a dataset. It costs little or nothing, and it is the only way to find out whether a provider's description matches reality for your use case. Yet many buyers glance at a sample file, see that the columns look right and move on. This playbook shows how to request a useful sample and test it properly.

What a good sample looks like

A useful sample has four properties:

• Representative: it reflects the full dataset, not a hand-picked showcase.
• Relevant: it covers your scope, meaning your countries, industries or entities.
• Complete in structure: it contains every field in the full product, in the delivery format you will receive.
• Documented: it comes with a data dictionary, field definitions and a note on how it was drawn.

A sample that fails any of these can still be informative, but you should know which conclusions it supports.

Ask for the right sample

Your entities or theirs?

The best test is a sample built on entities you already know. Send the provider a list of companies, products, places or tickers you have verified data for, and ask them to return their records for that list. You can then measure both coverage (how many they found) and accuracy (how many are right).

If that is not possible, ask for a random sample from your scope and an explanation of how it was drawn.

How big should it be?

Big enough to measure what matters. For accuracy checks, 50 to 200 records you can verify by hand is usually enough to spot serious problems. For coverage and fill-rate checks, a few thousand rows give more stable results. For time series, ask for enough history to see at least one seasonal cycle.

Ask for the documentation too

Request the data dictionary, a description of collection methods, the update schedule and any known limitations. Testing data without definitions leads to false alarms, such as treating a revenue band as an exact figure.

The tests

Run these tests on every shortlisted provider, in the same way, so the results are comparable.

1. Structure and schema

• Do the columns match the documentation and the listing?
• Are types consistent (dates as dates, codes as codes)?
• Are identifiers present and stable, such as company registration numbers, domains, ISO country codes or ticker symbols?

2. Coverage or match rate

Of your test entities, what share appear in the sample? Break the result down by segment. A provider might cover large companies well and small ones poorly, or one country well and its neighbour badly.

3. Fill rate

For each must-have field, calculate the share of rows where it is populated with a meaningful value. Watch for placeholders such as "N/A", zeros or default dates that inflate fill rates.

4. Accuracy against ground truth

Check a subset by hand against a trusted source: your CRM, official registries, a company's own website, or a physical check for places. Record each field as correct, incorrect or unverifiable. Accuracy is the share of verifiable values that are correct.

5. Freshness

Look at last-updated or last-verified dates and compare them with what you know. If a company moved offices or a shop closed recently, does the sample reflect it?

6. Consistency over time

For time series, look for gaps, sudden level shifts and duplicated periods. Ask whether historical values are point-in-time or restated.

7. Duplicates and entity resolution

Search for the same entity appearing under slightly different names or identifiers. Poor entity resolution causes double counting and messy joins.

A worked example

Suppose a sales team wants firmographic data for 2,000 target accounts in Spain and Italy. They send 300 known accounts to three providers and receive a sample back from each.

• Provider A matched 270 of 300 accounts (90%), with a 95% fill rate on employee count and headquarters city.
• Provider B matched 240 of 300 (80%) but had the most accurate industry codes on a hand-checked set of 100.
• Provider C matched 285 of 300 (95%), but a hand check showed many employee counts several years out of date.

None of these is automatically the winner. If the team mainly needs industry segmentation, Provider B may be best despite lower coverage. If it needs headcount to route accounts, Provider C's freshness problem is serious. The point of the test is to make that trade-off visible. The figures here are illustrative; your own results will differ.

Scoring and deciding

Summarise results in a small table: match rate, fill rate for each must-have field, accuracy on the hand-checked subset and freshness. Weight them according to the decision the data supports, then compare with price. A cheaper dataset that needs heavy cleaning may cost more in total.

If results are close, ask each finalist about the weakest area you found. Their explanation, and whether they offer to fix it, tells you a lot about the support you will get after signing.

Common pitfalls

• Testing only the showcase sample. Curated samples are useful for checking structure, not quality.
• Comparing different samples. Use the same test entities and scope for every provider.
• Confusing fill rate with accuracy. A field can be fully populated and still wrong.
• Ignoring licence terms on samples. Some samples may only be used for evaluation and must be deleted afterwards.
• Stopping at the sample. Rerun the same checks on the first full delivery.

Requesting samples on DataStoa

Many listings show preview rows you can inspect before contacting anyone, and providers flag when a free sample is available. To ask for a tailored sample, contact the provider from the listing and choose a sample request, describing your scope and test entities. If you are comparing several providers, post a data request that asks for a sample with each proposal. You can also search for listings that offer free samples.

Frequently asked questions

Are data samples free?

Usually, yes. Most providers offer a free sample for evaluation. Large custom samples, such as a full historical extract or a match on tens of thousands of your records, may be charged or credited against a later purchase.

Can I share my customer list with a provider for a match test?

Share only what is needed, such as company names and domains rather than personal data, and put a non-disclosure or data processing agreement in place first. Ask the provider to delete your test list after the evaluation.

What if the sample contains personal data?

Treat it like any other personal data: limit access, use it only for evaluation and delete it when you are done. Check that the provider can explain the legal basis for holding it.

How do I know a sample is representative?

Ask how it was drawn and, where possible, supply your own test entities. Compare a random subset with a selected subset if the provider sends both.