New binning strategy for bitmap indices on high cardinality attributes

Abstract

Bitmap indices are the preferred indexing structures for read only & high dimensional data in data warehouses and scientific databases. High cardinality attributes pose a new challenge in terms of having space efficient bitmap indices. Binning is a common technique for reducing space requirements of bitmap indices. It is found that binning has an adverse affect on the query performance. A new efficient binning strategy is proposed for bitmap indices for high cardinality attributes. Exact bins are created based on query distribution. Exact bins are allowed to overlap. This gives a considerable performance advantage over the conventional non-overlapping bins at the expense of marginal increase in space overheads. Overlapping bins minimize the number of candidate-checks that need to be performed for a given set of queries. Algorithms are also presented for performing candidate checks more efficiently. Experimental results are presented in support of the new binning strategy.