The Hindu Business Line : Statistical significance

There are lies, damn lies, and then statistics,” is a remark attributed — probably wrongly — to Benjamin Disraeli and subsequently popularised by Mark Twain. Essentially, it refers to the use of statistical analysis and data to bolster a weak argument, or to further a case that has no other justification. Naturally, such a tactic can succeed only if other people don’t understand the statistics well.

Consider one of the key aspects of statistical analysis — significance testing. Let me try and explain this without using mathematics at all.

Whenever we measure anything through a sample survey, such as the percentage of people who use a particular brand of soap or the average amount of money spent on eating out each month, the survey finding will be somewhat accurate but not exactly so. There are formulae to help us determine the likely level of accuracy(note: likely level of accuracy). One of the key elements in such formulae is the sample size — the bigger the sample size, the greater is the likelihood that the accuracy level is good. However, the point is that the figure measured through a sample survey is inaccurate.

Let us assume Scenario 1 where we measure the awareness for a brand through a sample survey, and then do it again after six months. Since each figure will have some degree of error associated with it, the two figures will probably not be the same, even though nothing has happened in the interim to actually affect brand awareness.

Let us now assume Scenario 2 where we measure the awareness for a brand through a sample survey and then run an ad campaign. At the end of the campaign, we once again measure the brand awareness with another sample survey. Again, the two figures will not be the same. However, the difference could be either because of sampling error or because an ad campaign has been run.

This is where significance testing comes in useful. If we run an appropriate test (for difference in proportions, in this example), we will be able to calculate how likely it is that the difference is because of the campaign ad and not simply because of sampling error. We can then conclude whether the campaign had any impact or not.

This is also where a note of caution has to be struck when it comes to interpreting the results. If the significance test says that the difference is probably because of the campaign that does not automatically mean that the campaign is a success. The campaign can be treated as a success when the extent of difference justifies the expense behind the campaign. Significance testing cannot answer the question of whether the difference is large enough or not.

To gain a better understanding of this subject, the best reference book is probably the statistics textbook written by Prof. P. K. Viswanathan. For courses on statistics, The Market Research School can be contacted ( www.tmrs.in)

(Contributed by Ashok R. Sankethi, CEO, Kaybase, a business consulting firm. Mail: ashok@kaybase.com)

More Stories on : Marketing Research | Insight | Brands | Capsule

Article E-Mail :: Comment :: Syndication :: Printer Friendly Page

Stories in this Section
The tyranny of ‘or’

Angry young managers

Statistical significance