Fuzzy Membership Partition Based Effective Hierarchical Agglomerative Flat Clustering Method for High Dimensional Data

Abstract

Introduction: Hierarchical clustering is an unsupervised powerful method for empirical knowledge interpretation from data. It has a fundamental role in understanding the complex pattern in huge datasets. It creates a hierarchical representation of data by forming clusters in two ways namely Agglomerative (Bottom-up) and Divisive (Top-Down). The main advantage is that it does not need to fix number of clusters. 
Objectives: To handle the issues such as, the pertinence for enormous data is minimal as the computational complexity is high in using Hierarchical clustering, complication of fixing Threshold value in Dendrogram height while combining Flat clustering, and non existence of mathematical objective function to assess the Hierarchical clustering. 
Methods: On focusing on these challenges, this work proposes (a) a liner split of data in order to reduce the computational complexity in Hierarchical Agglomerative clustering. (b) Fuzzy Partition matrix is created to enhance the cluster generation in Hierarchical clustering. (c) this work applies an objective function in Flat Clustering to ease the process of fixing threshold.
Implementation, Results:This work is implemented in Rapidminer tool. The Sum of Squares, Cluster Density and Processing Time is minimized in th e proposed work.
Conclusions:  The proposed method handles enormous data effectively using linear split with sequential execution, the proposed usage of a sum of squares fixes a optimum threshold value in dendrogram height while transforming the dendrogram to flat clusters, the proposed method improves the existing Hierarchical clustering effectively.