Multilevel Event Detection, Storyline Generation, and Summarization for Tweet Streams

Abstract

Users acting as real-time sensors post information about current events on various social media sites like Twitter, Facebook, Instagram, and so on. This generates a huge amount of data requiring significant effort to process and filter it to detect events/topics. It becomes more challenging when data are generated as a tweet stream because of its speed, presence of noise, slangs, phrases, abbreviations, and so on. In recent years, many approaches have been proposed either for detecting small- or large-scale events, individually. There is a lack of a complete solution that provides analysis from different perspectives. We propose a novel approach Mythos that detects events, subevents within an event, and generates abstract summary and storyline to provide different perspectives for an event. There are three modules in Mythos. Online incremental clustering algorithm identifies small-scale events in the form of small clusters, the event hierarchy generator generates bigger events in the form of hierarchies, and the summarization module produces summary of events/subevents. The summarization module uses a long short-term memory (LSTM)-based learning model to generate summaries at different levels-from the most abstracted to the most detailed. The summaries at different levels are used to generate a storyline for the event. Our experimental analysis on a variety of twitter data sets from different domains compares Mythos against the known existing approaches for event detection and summarization. It outperforms baseline approaches for both. The generated summaries are evaluated against summaries provided by external reference sources like Guardian and Wikipedia.