Variety management for big data
pp. 47-62
in: Thomas Hoppe, Bernhard Humm, Anatol Reibold (eds), Semantic applications, Berlin, Springer, 2018Abstract
Of the core challenges originally associated with Big Data, namely Volume, Velocity, and Variety, the Variety aspect is the one that is least addressed by the standard analytics architectures. In this chapter, we analyze types and sources of variety and describe data- and metadata management principles for organizing data lakes. We discuss how semantic metadata can help describe and manage variety in structure, provenance, visibility and permitted use. Moreover, ontologies and metadata catalogs can aid discovery, navigation, exploration, and interpretation of heterogeneous data lakes, and can simplify interpretation, lift data quality, and simplify integration of multiple data sets. We present an application of these principles in a data architecture for the Law Enforcement domain in Australia.