3 Eliciting Measures

Required reading:

Chapter 6, Pragmatic Social Measurement

3.1 Seminar

For this assignment, we are going to look at measures that relate to specific occupations (jobs). Given the topic this week, we are going to focus on measures that arise through processes of direct elicitation. The target concepts, for the purposes of this exercise are (1) how prestigious different occupations are and (2) whether different occupations should be regulated (in the sense that, in order to carry them out or to claim them as a title, one should be legally required to have certain professional qualifications or experience). Both of these are concepts where there is not an obvious gold standard that is independent of existing beliefs and practice, and so they provide an opportunity to think about different approaches to validation.

The two concepts also allow us to practice the different approaches required to assess validity and reliability in interval-level data (occupational prestige) and binary categorical data (occupations requiring regulation).

This assignment is partially derived from results reported in a paper entitled “Occupational prestige and occupational social value in the United Kingdom: New indices for the modern British economy” by Gemma Newlands and Christoph Lutz (2024). In their study, they asked UK survey respondents the following question:

For each listed occupation below, please use the slider to indicate how you would rate the prestige of the occupation on a scale of 0 (the lowest level of prestige) to 100 (the highest level of prestige).

They report the mean and standard deviation for each of large number of occupational categories, which are defined by International Standard Classification of Occupations ISCO-08 occupational codes.

The UK government describes “regulated professions” as follows:

Some professions in the UK are regulated. This can mean there is a legal requirement to have certain qualifications or experience. Other professions are regulated by chartered bodies granting a certain status.

If a profession is regulated, you may need certain professional qualifications or experience to:

carry out certain professional activities

use a professional title, like ‘architect’

For many professions, the bodies that decide if you are qualified to work in a regulated profession are known as ‘regulators’. Some professions are also regulated through chartered professional bodies.

In this exercise, we are going to look at data collected from various LLMs in which they (1) rate the occupational prestige of a selected set of occupations and (2) code whether they think that these occupations ought to be “regulated professions”. We will consider the results of the prestige survey by Newlands and Lutz and the actually regulated professions in the UK as benchmarks for LLM responses, and think about what these kinds of benchmarks can and cannot tell us about the reliability and validity of the elicited measures.

The main data file for this exercise includes the average value op_mean from the Newlands and Lutz survey data, and a set of variables

prestige_gemini_3.5_flash_lite
prestige_gemini_3.6_flash
prestige_gemini_3.1_pro
prestige_chatgpt
prestige_chatgpt_thinking
prestige_claude_haiku_4.5
prestige_claude_sonnet_5.0
prestige_claude_opus_5.0

that resulted from asking various LLMs to rate the prestige of the listed occupations on a 0 to 100 scale.

The data file also has codings of whether the same occupations ought to be legally regulated:

regulation_gemini_3.5_flash_lite
regulation_gemini_3.6_flash
regulation_gemini_3.1_pro
regulation_chatgpt
regulation_chatgpt_thinking
regulation_claude_haiku_4.5
regulation_claude_sonnet_5.0
regulation_claude_opus_5.0

Note that this coding task is not part of the Newlands and Lutz data, so there is no population survey benchmark for this part of the exercise. However, there is a variable in the data set regulated which indicates whether these occupations are actually regulated in the UK (there are some ambiguous cases, see ambiguous and note variables for details, but we will ignore this for the purposes of the assignment).

Open the data file week-3-data-entry-template.csv that you downloaded above in Excel or in a basic text editor. It has rows for each of 20 occupational categories, and two empty columns prestige_my_rating and regulation_my_coding. Please take a couple minutes to go through each of the 20 occupational categories, entering your personal view as to the level of occupational prestige and whether the profession ought to be regulated in the sense described above. When you are done, save the data keeping the same name and csv file format. Then load both this data file and the other data file into R and then merge them. For example, if the files are in the current R working directory, you could load and merge these as follows:

my_data <- read.csv("week-3-data-entry-template.csv")
occ <- read.csv("week-3-occupation.csv")
df <- merge(my_data,occ)

Compare your occupational prestige ratings that you saved in prestige_my_rating to the average ratings from the Newlands and Lutz survey data op_mean, treating the latter as a gold standard. On this basis, calculate the average measurement error (bias), the standard deviation (variance) of the measurement errors, and the reliability (calculated as \(\rho^2\), where \(\rho\) is the Pearson correlation) of the measure.

Do the same calculations for each of the LLM measures, again using op_mean as the gold standard measure, and organise these into a table with rows for each LLM, and columns for bias, variance (expressed as a standard deviation) and reliability (expressed as \(\rho^2\)). What would you say in general about how the LLMs performed at this task? How did your own ratings perform relative to the LLMs?

Calculate the inter-coder reliabilities for the LLM measures against one another (again, as \(\rho^2\)). How do these compare to the reliabilities that you calculated against op_mean for each of those measures in the previous question? Are they larger or smaller in general? If these measures have independent errors by comparison to the gold standard, would we expect the intercoder reliabilities to be higher or lower than the reliabilities versus the gold standard? Are the measurement errors relative to the gold standard correlated across the different LLM measures?

Which occupations do the LLM coders rate more highly relative to the population survey benchmark, averaging across LLMs? Which occupations do the LLM coders rate less highly relative to the population survey benchmark, averaging across LLMs? Do you notice any patterns in which occupations the LLMs tend to rate more highly than the human benchmarks, and vice versa?

Make a scatterplot with the survey gold standard as the x-variable, and which shows all of the individual LLM measured values for the same occupations as the y variable. Plot these as small points, unlabeled. Also plot the LLM averages for each occupation, using larger points and with the occupation labeled. Are the average errors of the LLMs relative to the survey gold standard systematically different between occupations with low levels of prestige and high levels of prestige?

What are some reasons why it makes sense to use a population survey average as a gold standard benchmark for a concept like occupational prestige? Can you think of reasons it might not make sense to treat these data as a gold standard benchmark?

Now, let’s turn to the data on regulated professions/occupations. Examining the various regulation_[llmname] variables, and using the variable regulated as the gold standard, calculate the sensitivity and specificity of each measure against the gold standard. Remember, the sensitivity is the rate of measured positive cases among true positive cases; specificity is the rate of measured negative cases among true negative cases.

Which occupations are the LLMs disagreeing on? Which are they getting wrong relative to the gold standard? What might we conclude from this?

Review above how these measurements and the gold standard were constructed. How should this shape our interpretation of these patterns of the differences between them? What might be a more appropriate gold standard to use, if one could collect the appropriate data?

Required reading:

Further reading:

3.1 Seminar