Visualization Lab Class
Assignment 6
Text visualizations became more popular with the growing amounts of digitized, textual data. These kinds of visualizations explain relations between textual entities, abstract and summarize text passages and aim for giving users information about the text to help them find relevant text passages more quickly.
In this exercise, you will focus on the implementation and data preparation for word clouds.
Word Clouds
A Word Cloud is a nice, yet imprecise way to have a look at the textual contents of our data. To get an overview of the movies in our movie database, we will create one Word Cloud per genre based on the movie descriptions.
In a first step, you will implement the word cloud using the library d3-cloud. An example of how to use it can be found on Observable. As always, you are not allowed to copy the function, but you can take inspiration from it and adapt it to your needs.
Task 1: Word Cloud Implementation (40%)
Implement the missing parts of the word cloud implementation in the wordcloud.js.
Your implementation will automatically be feed with the fake data given in the
index.js, such that you can test. The fake data will be replaced in Task 3
with real data from film descriptions.
Word Vectors
In order to create most text-based visualizations, first we need to transform each input document - the movie's description in our case - to a word vector.
There are several ways to define a word vector. In the lecture, you have been introduced to the sparse version. To safe the browsers internal memory, we will use the dense representation. In this representation, the word and its value is stored. Let's consider the following example of a well known film quote: "Bond. James Bond.". The dense word vector using word count as value would look like this: [["Bond", 2], ["James", 1]], since the term Bond figures two times in the quote, while James occurs only once.
The word count is a simple measure, but it emphasizes common words like and since these words have a high word count. To mitigate that, we will use the tf-idf score instead. The score expresses, how much a term helps to distinguish one document from any other document.
In the tf-idf score, tf stands for term frequency and is the word count
weighted by the total number of words in the document, aka movie description:
f is the word count of a term t in document d.
idf stands for the inverse document frequency and is calculated
by counting the number of documents, the term occurs in, weight it by the overall number
of documents and invert the result. To ensure that frequent terms are weighted less, usually
the log of the inverse document frequency is used:
where N is the total number of documents in corpus D and t is again the term/word.
The tf-idf is then calculated by multiplying tf and idf:
The formulas are taken from the
tf-idf page on Wikipedia.
I have also prepared an Observable page that shows an
example of how to calculate the tf-idf scores.
Task 2: Word Vectors (40%)
Transform each of the given movie's descriptions to a dense word vector
using the tf-idf score. For this, first calculate all idf values
by completing the inverseDocumentFrequency function in the
wordvector.jsfile. It will fill a map, in which the idf
value for each term of the corpus is stored.
In a second step complete the tfidf function in the
wordvector.jsfile, which
is supposed to transform a movie description, that has already been broken
into single words,
to a dense word vector.
Once the tf-idf score is implemented, the visualization below shows the 15 words with the highest tf-idf scores for 10 movies. For now it shows the first 15 words in the movie's description. You can use it to check your implementation.
If you like to compare numbers, this would be the first 5 entries of the correct word vector of the film Four Rooms: [ [ "bellhop", 0.43 ], [ "predicaments", 0.43 ], [ "unbelievable", 0.39 ], [ "evening", 0.31 ], [ "ted", 0.31 ] ]
Word Cloud Preparation
Now, lets connect Task 1 and Task 2! Using the function for tf-idf in combination with the word cloud function, we can now explore the most frequent words used in the descriptions of films per genre.
Task 3: Word Cloud preparation (20%)
Calculate the tf-idf scores for all film descriptions per genre in
the main.js.
For this, first generate a word array for each genre by concatenating the word arrays of that genre's movies.
Then calculate first the idf values of all words of using the genre word arrays and finally the word vectors with tf-idf values.
To visualize the data, insert your calculated data into the
wordcloud function in the index.js.
The list of genres is
already prepared for you in the main.js.