In this chapter, the authors discuss the characteristics of data collected by the New Zealand Centre for Adverse Drug Reaction Monitoring (CARM) over a five-year period. The authors begin by noting the ways in which adverse reaction data are similar to market basket data, and the ways in which they are different. They go on to develop a model for estimating the amount of missing data in the dataset, and another to decide whether a drug is rare simply because it was only available for a short time. They also discuss the notion of “rarity” with respect to drugs, and with respect to reactions. Although the discussion is confined to the CARM data, the models and techniques presented here are useful to anyone who is about to embark on an association mining project, or who needs to interpret association rules in the context of a particular database.
As I was going up the stair
I saw a man who wasn’t there.
He wasn’t there again today
I wish, I wish he’d go away.
Antigonish, William Hughes Mearnes, 1899.
TopMarket Baskets Vs. Adverse Reactions
Let’s start by considering how we get our cases in market basket analysis.
- 1.
A shopper decides to buy some items on
- 2.
a shopping list and makes
- 3.
a shopping trip to get them. During this trip,
- 4.
s/he enters a shop,
- 5.
selects some items,
- 6.
is unable to locate others,
- 7.
and chooses others on impulse.
- 8.
S/he pays at the cash register and leaves.
- 9.
The register record is later analysed.
There are demographic facts about the shopper like age, sex, and home district that are normally missing from the cash register record. It is not that such data are not available in principle, it is that they are not measured or recorded. Loyalty cards may permit such information to be gathered. Researchers might still not have access to it for privacy reasons.
Market basket analysis is sometimes presented as giving us information about shoppers. Without loyalty card or banking information to link one shopping episode to another, what we actually have is information about trips, not people. If someone habitually shops for food on Mondays and cleaning products on Wednesdays, we will never learn about associations between food products and cleaning products for that shopper.
We do not learn about people who wish to buy goods we have for sale but who never enter the shop, or who enter and leave without making any purchase. Perhaps they do not like the lighting, or the scents of the cleaning products make them feel sick. (Such scents are often unpleasantly strong.)