Catur Approach to Assess the Quality of Big Data Using Decision Tree and Multidimensional Model

Abstract

This paper is intended to design and develop multidimensional and decision tree based frameworks, for assessing the quality of a big data. Since the datasets represented in a big data environment is both complex and multidimensional, the quality of big data can be better viewed through multiple dimensions. Most enterprises face number of challenges in managing the quality of the big data during their initial setup or migration from traditional database or after building the big data. This paper uses multidimensional model proposed for Knowledge Management System for designing critical quality dimensions for big data. Based on the extensive literature review, this work proposes a classification of big data quality into many quality factors such as accessibility, consistency, integrity, usability, relevance, completeness, compatibility, conformity and accuracy. Since there are very few appropriate data stewards or frameworks available for confirmation of quality dimensions, this paper aims to develop some hybrid approaches using multi-dimensional model and decision tree based methods for automatic quality checks. Using decision tree, multiple if-then rules can be formed to decide on the quality of data based on the specific constraints developed for big data. The paper also aims to provide the quality framework and measures which can serve as a data quality firewall just like an internet firewall to proactively find the quality issues and apply the rules based on the decision tree algorithms to prevent bad or inconsistent or invalid data or access entering in to the big data environment.