7  Word Embeddings

7.1 Similarities, Analogies and Dictionary Expansion

Word embeddings are all the rage.1 Whenever we have represented words as data in previous weeks, we have simply counted how often they occur across and within documents. We have largely viewed words as individual units: strings of text that uniquely identify a given meaning, for which we have had no natural notion of similarity for grouping similar words together.

1 A good qualitative indicator of the success of an innovation in quantitative methods is when they are discussed in some detail in the London Review of Books

By contrast, word-embedding approaches represent each unique word in a corpus as dense, real-valued vector. As we discussed in the lecture, these vectors turn out to encode a lot of information about the ways in which words are used, and this information can put to good use across a wide range of questions in the social sciences.

In the seminar today, we will familiarise ourselves with some of the pre-trained word embeddings from the GloVe project. We will use these vectors to discover similarities between words, to compute analogy-based tasks, and to supplement the dictionary-based approaches to measurement that we covered in week 2.

7.2 Packages

You will need to load the following packages before beginning the assignment

library(tidyverse)
library(quanteda)
library(text2vec)
# If you cannot load these libraries, try installing them first. E.g.: 
# install.packages("text2vec")

7.3 Data

Today we will be using the pre-trained GloVe embeddings, which can be downloaded from the link at the top of the seminar page. Note that the file which contains the word embeddings is very large! It may therefore take a minute or two to download, depending on your internet conection.

Despite the large size of the file, we are actually using one of the smaller versions of the GloVe embeddings, which were trained on a combination of Wikipedia and news data. The embeddings are of dimension 300 and cover some 400,000 words. Note that you could replicate any of the assignment here with larger versions of the GloVe embeddings by downloading them from the GloVe project website, but any differences for the applications here are likely to be small.

7.4 Word Similarities

  1. Load the glove embeddings into R using the load() function.
Reveal code 
load("glove_embeddings.Rdata")
  1. Look at the dimensions of the glove embeddings object. How many rows does this object have? How many columns? What do these represent?
Reveal code 
dim(glove)
[1] 400000    300

Rows represent words. Columns represet the dimensions of the embedding.

  1. Write a function to calculate the cosine similarity between a selected word and every other word in the glove embeddings object.2 It will be useful for you to use the sim2() function3 from the text2vec package here. Your function should take two inputs:

    1. target_word – the word for which you would like to calculate similarities
    2. n – the number of nearest neighbouring words returned

2 In contrast to previous weeks, I have not given you the full solution here. Instead, I have provided some starter code below. You should be able to work out what goes in each part of this function by looking at the examples in the lecture slides. I will release the full code when I release the homework solutions at the end of the week.

3 Note that this function requires two main arguments: 1) x – a matrix of embeddings. 2) y – a second matrix of embeddings for which you would like to compute similarities. It is important to note that both of these inputs must be in matrix form. As the mean vector embedding that we created above is actually a numeric vector, we have to transform it to a matrix that has the same number of columns as the glove embeddings object. To do so, use the matrix function, setting the nrow argument equal to 1.

Reveal code 
similarities <- function(target_word, n){

  # Extract embedding of target word
  
  # Calculate cosine similarity between target word and other words
  
  # Report nearest neighbours of target word
  
}
  1. On the basis of the Glove word embeddings, what are the 7 most similar words to the words “quantitative”, “text”, and “analysis”?

7.5 Word Analogies

  1. Write a function that computes analogies of the form “a is to b as c is to ___”. For instance, if b is”king”, a is “man”, and c is “woman”, then the missing word should be “queen”.
    Your function will need to take four arguments. Three arguments should correspond to the words included in the analogy. The fourth should be an argument specifying the number of nearest neighbouring words returned. Again, I have provided some starter code below, which you should be able to complete by consulting the lecture slides.
Reveal code 
analogies <- function(a, b, c, n){
  
  # Extract vectors for each of the three words in analogy task

  # Generate analogy vector (vector(c) - vector(a) + vector(b))
  
  # Calculate cosine similarity between anaology vector and all other vectors
  
  # Report nearest neighbours of analogy vector

}

  1. Use the function you created above to find the word-embedding answers to the following analogy completion tasks.

    • Einstein is to scientist as Picasso is to ___?
    • Arsenal is to football as Yankees is to ___?
    • Actor is to theatre as doctor is to ___?
  1. Come up with some of your own analogies and try them here.

7.6 Dictionary Expansion

In seminar 2 we used the Moral Foundations Dictionary to score some Reddit posts in terms of their moral content. In this exercise we will expand the “care” category of the MFD using the Glove embeddings.

For this part of the assignment, you will need to have access to the files that we used in seminar 2. These were mft_dictionary.csv and mft_texts.csv. Go and locate them now (or redownload them if you need them).

Once you have found these files, load them into R:

mft_dictionary_words <- read_csv("mft_dictionary.csv")
mft_texts <- read_csv("mft_texts.csv")
  1. Create a vector of the MFT “Care” words.
Reveal code 
care_words <- mft_dictionary_words$word[mft_dictionary_words$foundation == "care"]
  1. Extract the embeddings from the glove object relating to the care words.
Reveal code 
care_embeddings <- glove[rownames(glove) %in% care_words,]
  1. Calculate the mean embedding vector of the care words. To do this, use the colMeans function, which will calculate the mean of each column of the matrix.
Reveal code 
care_embeddings_mean <- colMeans(care_embeddings)
  1. Calculate the similarity between the mean care vector and every other word in the glove embedding object. To do so, use the sim2() function again.
Reveal code 
target_sim <- sim2(x = glove,
                   y = matrix(care_embeddings_mean, nrow = 1))
  1. What are the 500 words that have highest cosine similarity with the mean care vector? How many of these words are in the original dictionary?
Reveal code 
top500 <- names(sort(target_sim[,1], decreasing = T))[1:500]

table(top500%in%care_words)

FALSE  TRUE 
  386   114
  1. Examine the words that are in the top 500 words that you calculated above but which are not in the original care dictionary. Do these represent the concept of care?
Reveal code 
top500[!top500%in%care_words]
  [1] "traumatized"     "victimized"      "cruelly"         "helpless"       
  [5] "innocent"        "unborn"          "sufferings"      "endure"         
  [9] "terrified"       "frightened"      "sick"            "callous"        
 [13] "wronged"         "civilians"       "viciously"       "senseless"      
 [17] "terrorizing"     "frighten"        "beatings"        "horrific"       
 [21] "risking"         "inhumane"        "unspeakable"     "confess"        
 [25] "humiliation"     "defenseless"     "loneliness"      "terrorize"      
 [29] "grief"           "bereaved"        "injustice"       "subjecting"     
 [33] "vengeful"        "innocents"       "misfortune"      "abuse"          
 [37] "neglect"         "plight"          "treating"        "traumatised"    
 [41] "stigmatized"     "oppression"      "starving"        "humiliate"      
 [45] "oftentimes"      "grievous"        "unjustly"        "grieving"       
 [49] "humiliated"      "betrayed"        "spared"          "mutilation"     
 [53] "lest"            "ashamed"         "heinous"         "enemies"        
 [57] "abusive"         "barbaric"        "elderly"         "starve"         
 [61] "fear"            "neglecting"      "terrible"        "deprived"       
 [65] "dignity"         "deprive"         "beings"          "dying"          
 [69] "mercilessly"     "hatred"          "grievously"      "oneself"        
 [73] "vicious"         "succumb"         "feelings"        "schoolmates"    
 [77] "fearful"         "horrible"        "horrifying"      "sparing"        
 [81] "hardships"       "afraid"          "risked"          "indiscriminate" 
 [85] "unnecessarily"   "bystanders"      "heartless"       "shame"          
 [89] "ordeal"          "offends"         "subjected"       "enslavement"    
 [93] "handicapped"     "disturbed"       "humanity"        "sacrificing"    
 [97] "degrading"       "oppressed"       "treat"           "retribution"    
[101] "maiming"         "injure"          "terrify"         "terminally"     
[105] "unbearable"      "betray"          "suicidal"        "pretending"     
[109] "seriously"       "perpetrated"     "destitute"       "maimed"         
[113] "perpetrator"     "insult"          "demeaning"       "depraved"       
[117] "betraying"       "selfishness"     "aiding"          "traumas"        
[121] "taunt"           "brutal"          "belittling"      "sickening"      
[125] "despair"         "needless"        "orphans"         "repress"        
[129] "terrorized"      "hardship"        "scared"          "infirm"         
[133] "husbands"        "savagery"        "townspeople"     "mentally"       
[137] "punishing"       "disciplining"    "sins"            "savagely"       
[141] "escaping"        "teasing"         "sickness"        "distraught"     
[145] "captors"         "pregnant"        "oppress"         "spouse"         
[149] "indignity"       "intolerable"     "intimidated"     "forgiveness"    
[153] "countrymen"      "cursed"          "conscience"      "knowing"        
[157] "trauma"          "deserve"         "punishes"        "strangling"     
[161] "wickedness"      "sinful"          "punished"        "cruelties"      
[165] "helplessness"    "punish"          "disrespect"      "perceived"      
[169] "appalling"       "hysterical"      "bystander"       "disfigurement"  
[173] "treachery"       "homosexuals"     "debilitating"    "evils"          
[177] "taunts"          "vindictive"      "betrayal"        "tolerate"       
[181] "provokes"        "thoughtless"     "avenge"          "intimidate"     
[185] "feel"            "offend"          "despicable"      "needlessly"     
[189] "malnourished"    "sexually"        "tolerating"      "bodily"         
[193] "terrifying"      "passivity"       "abduct"          "betrays"        
[197] "trusting"        "frightening"     "expose"          "misguided"      
[201] "barbarity"       "severely"        "perceive"        "imprison"       
[205] "horrendous"      "depriving"       "sadistic"        "committing"     
[209] "unworthy"        "treated"         "unjust"          "slaughtering"   
[213] "painful"         "debilitated"     "scolding"        "enslave"        
[217] "aftereffects"    "disrespectful"   "victimised"      "awaken"         
[221] "humankind"       "resentful"       "suffocate"       "hypocritical"   
[225] "cowardly"        "fleeing"         "judgmental"      "ridicule"       
[229] "insecurities"    "complicit"       "misery"          "caretakers"     
[233] "detriment"       "physically"      "tenderly"        "habitually"     
[237] "malnutrition"    "punishment"      "misbehaving"     "villagers"      
[241] "emotionally"     "relatives"       "selfish"         "deserving"      
[245] "destitution"     "gravely"         "exposing"        "companionship"  
[249] "squeamish"       "brutally"        "guilt"           "imprisoning"    
[253] "orphaned"        "criminals"       "injustices"      "adolescents"    
[257] "stigma"          "disobedient"     "sacrificed"      "survivors"      
[261] "remorse"         "empathetic"      "wanton"          "insults"        
[265] "unsuspecting"    "enslaving"       "aggression"      "involuntarily"  
[269] "outweighs"       "ignorance"       "screams"         "addicted"       
[273] "ostracized"      "subordinates"    "immoral"         "caregivers"     
[277] "merciless"       "perpetrators"    "powerlessness"   "mutilating"     
[281] "ills"            "depredations"    "stab"            "affection"      
[285] "ill-treatment"   "ill"             "blinded"         "callously"      
[289] "insulted"        "manipulative"    "verbally"        "unintentionally"
[293] "distract"        "illness"         "ungrateful"      "deprivation"    
[297] "pretend"         "undermines"      "disfigured"      "impotent"       
[301] "cursing"         "inconvenience"   "motivates"       "witnessing"     
[305] "aggressors"      "jealousy"        "suffocating"     "children"       
[309] "sacrifices"      "jealous"         "hideous"         "ostracism"      
[313] "perverted"       "horrified"       "insensitive"     "confronting"    
[317] "illnesses"       "befriend"        "scourge"         "willful"        
[321] "repression"      "revenge"         "annoy"           "scarred"        
[325] "curses"          "throats"         "jailers"         "horribly"       
[329] "repressed"       "bigotry"         "excruciating"    "befriending"    
[333] "coerce"          "traumatic"       "ugliness"        "revulsion"      
[337] "exposes"         "patronizing"     "stepmother"      "newborns"       
[341] "indulging"       "promiscuous"     "confront"        "ministering"    
[345] "unconscionable"  "insidious"       "tenderness"      "indifference"   
[349] "gruesome"        "trampling"       "embittered"      "despise"        
[353] "cowardice"       "hopelessness"    "underlings"      "horrors"        
[357] "scold"           "atrocious"       "banish"          "persecutions"   
[361] "brainwashed"     "indignities"     "banishing"       "pretended"      
[365] "abhorrent"       "wrongs"          "genitals"        "counseled"      
[369] "atrocities"      "susceptible"     "ferocity"        "indifferent"    
[373] "autistic"        "deprivations"    "demoralizing"    "unfaithful"     
[377] "transgressions"  "brainwashing"    "ghastly"         "insufferable"   
[381] "taunting"        "pillage"         "believing"       "unconscious"    
[385] "afflicting"      "muggers"

It is relatively uncontroversial to suggest that these words are representative of the concept of “care”.

  1. What does your answer to the previous question suggest about this dictionary expansion approach?
Reveal code

The idea here is that using the word-embeddings has allowed us to automatically expand the set of of words associated with a concept that we previously measured using a dictionary of terms that were manually selected. This means that we can use information about word similarity to supplement our existing measurement approaches. Whether this leads to performance improvements in terms of classification is the subject of this week’s homework.

7.7 Homework

The mft_texts object includes a series of variables that record the human annotations of which category each text falls into. In this week’s homework, you will again use dictionary-based methods to score the texts, and compare the dictionary scores to those human codings. If you have forgotten how to apply dictionaries, go back and look at the material in seminar two.

  1. Create a new dictionary which includes two categories. The first should be a care_original_words category, which contains only the words from the original care dictionary. The second should be a care_embedding_words which contains both the original care words and the top 500 words that you extracted in the last section.
  1. Use the dictionary you just constructed to score the scores in the mft_texts object. Create variables in that object that indicate whether a given dictionary classifies each text as a care text or not (i.e. classify a text as a care text if it contains any words from the relevant dictionary).
  1. Create a confusion matrix which compares the human annotations to the scores generated by the dictionary analysis. Which performs best, the original dictionary or the word-embedding approach?

You should upload a paragraph explaining the results from the final question of the homework to this Moodle page.