Learn about data descriptions, data preprocessing, exploratory data analysis, k-means clustering, and more with Tableau 10’s newest clustering feature.
In this blog, we will discuss the clustering of customer activities for 24 hours by using the K-means clustering feature in Tableau 10. Tableau 10’s clustering feature automatically groups similar data points together. This type of clustering helps you create statistically based segments that provide insights about similarities in different groups and the performance of the groups when compared to each other.
You can use clustering on any type of visualization, ranging from scatter plots to text tables and even maps. In our previous blog post, Call Detail Record Analysis – K-Means Clustering With R, we discussed CDR analysis using the unsupervised K-means clustering algorithm.
A daily activity file from Dandelion API is used as a data source, where the file contains CDR records generated by the Telecom Italia cellular network over the city of Milano. The daily CDR activity file contains information for 10,000 grids about SMS in and out, call in and out, and Internet activity. This dataset has five million records and the size of the dataset is 314 MB. The below table, created in Tableau, shows the total activity of SMS, call, and Internet activity by hours and the total number of records per hour. The Grand Total section shows the cumulative total activity for SMS, call, and Internet.
To preprocess data, perform the following steps:
The calculation of the above new fields can be done in Tableau easily with Create Calculated Field features.
The below screenshot shows the formula used in Tableau for calculating activity_start_time. If the epoch time is in milliseconds, then divide the value by 1,000 to convert into seconds. The formulas for calculating other fields are as follows:
activity_date:
activity_hour:
total_activity:
total_sms_activity:
total_call_activity:
Tableau is bundled with rich sets of visualization to analyze the data. Exploratory data analysis is the process of analyzing the data visually. It involves outlier detection, anomaly detection, missing values detection, aggregating the values, and producing the meaningful insights. The following visualizations are created as part of EDA on five million data.
This visualization is used to find out:
From the above visualization, it is evident that most of the activities happened in the hour of 23 and much less activity happened in the hours of 5 and 6.
This visualization is used to find out:
From the above visualization, it is evident that most of the activities happened in the square grid ID 5059 and less activities happened in the square grid ID 497.
Let’s take a look at some clustering.
If the number of clusters is not specified by a user, Tableau picks the number of clusters corresponding to the first local maximum of the Calinski-Harabasz index. Tableau uses the following:
By default, K-means will be run for up to 25 clusters if the first local maximum of the index is not reached for a smaller value of K. A maximum value of 50 clusters can be set. To determine optimal number of clusters (K) , consider between-group sum of squares (SSB) , within group sum of squares (SSW) , and the total sum of squares from the cluster result.
Sometimes, data groupings make immediate sense. total_activity measure is used as a variable for clustering and K value is provided as 8. The Describe clusters dialog box provides information about the models that Tableau computed for clustering. These statistics can be used to assess clustering quality.
Analysis of Variance (ANOVA) is a collection of statistical models and associated procedures useful for analyzing variation within and between observations that have been partitioned into groups or clusters. In this use case, ANOVA is computed for the total_activity variable and the resulting analysis of variance table is used to determine total_activity variable effectiveness to distinguish clusters.
The Summary tab identifies the inputs used to generate the clusters and provides some statistics characterizing the clusters. The clusters are ranked using the Total Activity by Activity Hours cluster from more to less traffic as follows:
This cluster is based on total_sms_activity measure as the variable for clustering and K value provided as 8.
This cluster is based on sum (internet_traffic_activity) measure as the variable for clustering and K value provided as 8.
This cluster is based on sum (sms_in_activity) and sum (sms_out_activity) measures as the variable for clustering and K value provided as 6.
This cluster is based on sum (call_in_activity) and sum (call_out_activity) measures as the variable for clustering and K value provided as 6.
By using this clustering mechanism, you can find the clusters that are making more traffic to the telecom network in the measure of total activity. Similarly, you can obtain more information like square grid and country code information to understand the square grid likely creating more revenue and more traffic to the telecom network and to target high customers based on their geo-location.
GitHub location
