4 Supervised Scale Measurement I: Comparison Data

Topics: Scoring competition data. Rating transfer systems. Bradley-Terry models. Interpretation of latent variable models.

Required reading:

• Chapter 7, Pragmatic Social Measurement

Further reading:

Bradley-Terry Models

• Heather Turner and David Firth. “Bradley-Terry Models in R: The BradleyTerry2 Package”

Applications - Political Science

• Peter John Loewen, Daniel Rubenson and Arthur Spirling. “Testing the power of arguments in referendums: A Bradley–Terry approach” Electoral Studies. Volume 31, Issue 1, March 2012, Pages 212-221
• Zucco Jr, Cesar and Batista, Mariana and Power, Timothy J. “Measuring portfolio salience using the Bradley–Terry model: An illustration with data from Brazil” Research & Politics 2019 6:1.

Applications - Health/Medicine

• J. N. S. Matthews and K. P. Morris. “An Application of Bradley‐Terry‐Type Models to the Measurement of Pain” 1995.

Applications - Geography

• P. A. Longley and N. Wrigley. “Scaling Residential Preferences: A Methodological Note” Journal of Economic and Social Geography. 1984.

---

4.1 Seminar

For this assignment, we are going to replicate some of the analysis in the paper “Measuring portfolio salience using the Bradley–Terry model: An illustration with data from Brazil” by Zucco Jr, Batista, and Power (2019). The authors of that paper conducted surveys of serving Brazilian legislators as well as academic experts (political scientists in Brazil), in order to assess which cabinet positions were more highly valued. Respondents were given random pairs of ministries, and “were asked to choose the ministry they thought a typical politician would prefer to obtain for his or her party.”

The original data are available on Dataverse. We will be working with a cleaned-up versions of the legislator and expert survey data sets, which can all be be downloaded above.

While we discussed in lecture that it is possible to estimate these models using a logistic regression with a certain specification of the covariate data, we will use a package that sets up the data appropriately and provides useful summaries more easily. You will need to install and load the BradleyTerry2 R library.

install.packages("BradleyTerry2")

library(BradleyTerry2)

1. Use the function BTm() to fit the Bradley-Terry model on the comparisons made by legislators using the code below. Then use the BTabilities() function to examine the estimates for the relative appeal of different ministries. You might wish to sort the output using order() to make it easier to see which ministries have the highest and lowest Bradley-Terry estimates.

BTm_fit <- BTm(
  cbind(Win_1,Win_2), # the outcome 
  player1 = Ministry_1, player2 = Ministry_2, # the players
  data = legislator_data) # the data

2. Use the code below to merge the Bradley-Terry estimates with the ministry data. Then run a linear regression analysis predicting the Bradley-Terry “ability” (coefficients) using log(Budget) and log(Appointees). Interpret the results of this regression. What might it tell us about which kinds of ministries are more appealing to Brazilian legislators?

merged_data <- merge(ministry_data,
                     BTabilities(BTm_fit),
                     by.x="Ministry_English",by.y= "row.names")

3. Repeat the same series of analyses from Q1 and Q2, but using the expert survey data expert_data instead of the legislator survey data legislator_data. What is the same and what is different about the estimates?

4. Plot the Bradley-Terry estimates based on the expert survey data against those based on the legislator survey data. Use a square plot, the same range for x and y on the plot, and overlay the line with intercept 0 and slope 1 so that you can see any numerical differences between the estimates clearly. Also calculate the correlation coefficient between the two sets of estimates.

5. Calculate the proportion of comparisons that we would expect the most appealing ministry to win against the least appealing ministry given the estimates from the legislator survey and also given the estimates from the expert survey (these will not be the same ministries in both cases, the point here is to get a sense of how big the range of estimates is). Code Hint: If you are solving this “by hand”, it is useful to know that if \(\log \left(\frac{p}{1-p} \right) = x\) then \(p = \frac{exp(x)}{1 + exp(x)}\). Alternatively, you can work it out by having R calculate the relevant predicted probabilities / fitted values using predict().

6. Do you find the comparisons in Q3, Q4 and Q5 reassuring or concerning regarding the conclusions that you had in Q2? What is the same and what is different about the estimates from the two different surveys?

7. Come up with at least one novel application of a Bradley-Terry model to measure some concept that interests you. This should take the form of quantifying some body of common knowledge among some set of individuals with relevant expertise. What is the prompt? What is the concept you are trying to measure? What is the relevant population of experts?
   

   For example, one could survey students completing their Q-step degree with randomly generated pairwise comparisons of their 6 Q-step modules. The prompt might be: “Which of these two modules did you think was a better module?”. The concept that you would then be measuring is which modules Q-step students thought were better/worse. The population of experts would be Q-step students who have completed all six modules.

References

Zucco Jr, Cesar, Mariana Batista, and Timothy J Power. 2019. “Measuring Portfolio Salience Using the Bradley–Terry Model: An Illustration with Data from Brazil.” Research & Politics 6 (1): 2053168019832089.