How to Create Semantic Differential (EPA) Scales using Qualtrics
Written by Brent Curdy Graduate Student, Duke University (updated 21 October, 2014)
Introduction
To create an “EPA Language Dictionary,” researchers collect data using participant responses to semantic differential scales (hereafter called EPA scales). Data collection for the creation of these dictionaries typically consists of a survey containing pre-screen and control questions, demographic questions, and EPA scales. The survey itself may be paper-based, computer-based, or a combination of paper and computer components, such as a paper-based pre-screen and demographic portion following by a computerized version of the EPA scales. Given the large word lists commonly used in Affect Control Theory (ACT) research and dictionary creation efforts, recording data from paper-based surveys, especially those that use paper forms to collect the EPA scales, can be time and resource intensive, inexact, and prone to errors in data transcription. For this reason, David Heise developed a Java applet called Surveyor, a self-contained survey instrument for the purpose of collecting semantic differential data. Many studies have successfully employed Surveyor for their data collection efforts. In addition to ease of data management, Surveyor automatically randomizes the presentation of stimuli, as Heise recommends four dimensions of randomization: 1) random assignment to stimulus sets (i.e. word or concept lists), 2) stimuli are presented in a random order, 3) for each stimuli, the scales for Evaluation, Potency, and Activity are presented in a random order, and 4) the orientation of the “endpoints” of each scale are randomized (i.e, Good/Bad – Bad/Good) (Heise 2010).
Although Heise updated Surveyor several times, its development has not kept pace with the recent Java and other software updates. Moreover, Surveyor is not currently internet compatible. For this reason, Jonathan Morgan, a graduate student under Lynn Smith-Lovin at Duke University, has recently updated and improved Surveyor. However, at the time of this writing, the availability of the program is limited as Morgan works with the university to secure copyrighting for the new version of the program. Development of Surveyor remains ongoing, and Morgan intends to make the program internet-compatible so it may be used in online data collection efforts.
With Surveyor still under development, an alternative computer-based measurement tool is needed. The online survey tool, Qualtrics, is the most promising alternative explored so far. Dr. Kimberly Rogers, Dr. Tobias Schroder, and I have conducted two data collection efforts using EPA scales in Qualtrics that yielded useable data. There are two limitation to using Qualtrics, and one key measurement difference, the significance of which must be determined by the researcher. First, Qualtrics can automatically randomize stimuli along only the first three of the four randomization dimensions. This limitation did not result in detectably biased data in the studies conducted so far, but this may be due to the relatively short list of stimuli (only 30-40 stimuli per subject). Second, the space between the scale labels “Extremely” and “Infinitely” on the slider scale should be 1.3 times as wide as that between the other scale labels, with “Infinitely” and its hash mark at the far ends of the scale. In Surveyor, the scale labels correspond perfectly to their EPA integer values. That is; “Neutral” corresponds to 0, “Slightly” corresponds to 1 or -1, “Quite” corresponds to 2 or -2, Extremely corresponds to 3 or -3, and “Infinitely” corresponds to 4.3 or -4.3. In Qualtrics, “Neutral” corresponds to 0, but due to the even spacing of all scale labels, the label and corresponding values are slightly off. This means that EPA values collected for terms using Qualtrics cannot be reliably compared to EPA values collected using Surveyor without adjustment (i.e., between a -4.3/+4.3 scale and a -4.0/+4.0 scale). Heise (2010) explains that the purpose for the additional .3 used in the Surveyor scale. Again, the researcher must determine the theoretical importance of these psychometric limitations when evaluating Qualtrics as a measurement tool. These limitations aside, the main benefit to Qualtrics is that it is entirely internet-based, meaning that the survey can be taken on any device with internet access (including mobile devices) and it can be linked to online participant pools such as Amazon Mechanical Turk.
This instructional document will show you how to create a semantic differential scale (EPA scale) in Qualtrics. Another tutorial here shows you how to link your Qualtrics survey to Amazon Mechanical Turk. Note that these instructions are goal-specific; that is, they show only how to create EPA scales; these instructions do not show you how to create an instructional segment, demographic and other survey questions, manipulation checks, experimental conditions, or debriefing screens, although you are certainly able – and advised – to add these. Here is the final survey that was created using the instructions below.
How to Create a Semantic Differential Scale (EPA Scale) in Qualtrics Important note: The following instructions show you how to create a single semantic differential scale that can be copied (in two stages) to create as many scales as desired. Qualtrics autosaves edits; CONTROL+Z will not work to undo mistakes or re-edits. As we will see, it is easy to make small mistakes that might cause us to undo large portions of our work. However, by creating one good scale, then copying it, we save ourselves the painstaking task of creating the scales individually. I will point out delicate operations as we come to them.
Step 1: Familiarization So that the instructions will be clear, the figure below shows the layout of the Qualtrics Survey Editor Screen.
Figure 1: Survey Editor Screen
1: Question Options Bar 2: Active Survey Tabs 3: Survey Options Tabs 4: a Question (In this case, “Question #2) 5: Advanced Question Options
Step 2: Create a default slider question To start from scratch, click “Create Survey” in the Active Survey Tabs at the top of the screen. A popup will appear. You give your survey a “Survey Name,” but you may leave the “Folder” box empty. Then click “Create Survey.” You will now see the Survey Editor screen (see Figure 1), but there will be no questions yet. Click the green “Create a New Item” button, or click the down arrow and select “Slider” (toward the top, on the right). If you just clicked the button, a default, 3-choice question will be added to your survey. Click on the “Change Item Type” button at the top of the Question Options Bar (See Figure 1) and select “Slider”. There are three types of sliders: we want the default slider (called “Draggable Sliders”) if you click the dropdown menu on “Sliders.”
Figure 2: selecting “Draggable Slider”
Figure 3: The default look of the question after selecting “Slider” or “Draggable Slider”.
In the Question Options Bar, adjust the question’s settings to: Choices: 1 Labels: 9 Type: Sliders Grid Lines: 0 Min Value1: -4 Max Value: 4 Number of Decimals: 1 Options: select “Custom Start Position” and “Center Labels.” Unselect “Show Value” and “Not Applicable” Validation Options: This is up to you; you may allow respondents to skip question, force them to answer, or request for them to answer. Validation Type: select “None” (This is an advanced feature that we do not use here)
Step 3: Customize the slider labels After adjusting the settings in the Question Options Bar, your question should look like this:
Figure 4: Default appearance of a single EPA scale.
First, drag the slider so that it is under the center label (“Label 5″ in the above image). When you do so, Qualtrics may automatically select “Show value” under the Question Options Bar (even though you did not select it). In this case, you will see a number value to the right of the slider. If this happens, just unselect (again) “Show value” in the Question Options Bar. Now click on “Click to write Label 1″. Click the drop arrow and select “Rich Content Editor”. A popup will appear.
Figure 5: Select “Rich Content Editor”
Figure 6: Enter centered text in popup
There are two things that need to go into this popup: 1) The label for the scale point and 2) a bar ( | )2. Both the scale label (“Infinitely”, in the above image) and the bar must be centered, as shown. Repeat this process for each of the slider labels so that they read: Infinitely, Extremely, Quite, Slightly, Neutral, Slightly, Quite, Extremely, Infinitely. Don’t forget to put the bar under each label. Caution: If you click on a label and edit it without using the rich content editor, you will undo the rich content settings! (i.e., it will not be centered anymore). Note: after you create the labels, you may want to adjust the slider so that it is centered under the “Neutral” label. Finally, on the text to the left of the slider: “Click to write Choice 1″. Instead of this text, enter a single space. If you leave the text box empty, Qualtrics will refill it with “Click to write Choice 1″ and it will appear in your survey!
Step 4: Customize the Endpoints and Concepts The endpoints and the concept to be rated (in this tutorial, I will use the Concept Term “Mother”) must be placed in the Question Text Box. The spacing of the endpoints (at each end of the scale) and the concept (generally centered above the scale) is accomplished using the HTML editor. Click the question text field and navigate to the HTML View, as shown below.
Figure 7: HTML editor in the Question Text Box
1: The normal appearance of a question (red box is added) 2: The appearance of a question after clicking on “Click to write the question text” 3: The appearance of a question after clicking “HTML View” in the top right of the text field Caution: Qualtrics saves changes automatically once you click outside of the text box. Therefore, once you make edits, click outside the box to save them. If you click outside the box accidently, you must re-open the text box in HTML View before editing. If you type anything in the text box while you’re not in HTML view, Qualtrics will automatically convert the previous edits to that text box back to non-HTML code and you will have to start over. Once you are in HTML View, enter the following HTML code into the text box: (You will need to type this code, as I don’t yet know how to paste plain text using WordPress. Using the code as-is will make the text left-aligned, which isn’t correct.)
<div style=”float: left; width: 50px;”><span style=”font-family:arial,helvetica,sans-serif;”><strong>Bad</div>
<div style=”float: right; width: 50px;”><span style=”font-family:arial,helvetica,sans-serif;”><strong>Good</div>
<p> </p>
<div style=”text-align: center;”><strong><span style=”font-size:16px;”><span style=”font-family: arial,helvetica,sans-serif;”>Mother</span></span></strong></div>
<p> </p>
The question should now look like this:
Figure 8: Model Question
It is worth mentioning that the question will look different in the actual survey than it does in the Editor.
Step 5: Creating Evaluation, Potency, and Activity scales At this point, you have a single, complete scale, except for the JavaScript. To create additional scales, select the question, scroll to the bottom of the Question Options Bar, and select “Copy Question.” Every element of the question will be replicated3 and the new copy will appear below the original question. In this tutorial, we will create the “long format” of semantic differential scales used by Rogers and Schroeder (forthcoming). This format contains seven scales for each Concept Term (“Mother” in this case). However, the astute reader will easily “short format” that displays only three sliders – one for Evaluation, Potency, and Activity – for each Concept Term. The Seven Scale Format: Evaluation: Bad – Good, Awful – Nice Potency: Powerless – Powerful, Little – Big Activity: Slow – Fast, Quiet – Noisy, Inactive – Active First, copy the first question until you have seven copies.
Figure 9: Seven copies of the first question.
Go to each question text box (starting with the second question, since the first question is already done) and 1) change the endpoints to the appropriate terms and 2) remove duplicate Concept Terms from the page. Remember to use HTML View to make edits. To change the endpoints, simply change the words “bad” and “good” in the code to their appropriate terms. The “duplicate Concept Term” in the current example is the term “Mother” as it appears in question 2, 4, 6, and 7; the Concept Term should only appear in the first question of each set of scales. To remove the term, highlight the HTML lines containing the term and delete them.
Figure 10: How to delete duplicate Concept Terms.
Figure 11: Endpoints edited and duplicate Concept Term removed.
Not seen in Figure 11 are questions 5 (with the Concept Term) and questions 6 and 7 (without the Concept Term) with their appropriate endpoint terms. These question can be seen, however, in Figure
JavaScript: Now that we have a block of seven questions, we will add the JavaScript. To do this, click on each question containing the Concept Term (“Mother”), then click the Advanced Question Options button (the little gear on the left of the question). After you press it, select “Add JavaScript” from the drop down menu.
Figure 12: The appearance of the popup after clicking “Add JavaScript…” (see #5 in figure 1)
The code that customizes the slider question was written by Andrew Taylor. Paste the following code into the Javascript window:
Qualtrics.SurveyEngine.addOnload(function()
{
/*Place Your Javascript Below This Line*/
var table=document.getElementsByTagName('table');
for(var i=0;i<table.length;i++)
{
var tableClass=table[i].className;
if(tableClass.indexOf("slider") !== -1) {
var theadtbody=table[i].childNodes;
for (var j = 0; j < theadtbody.length; j++) {
if (theadtbody[j].tagName == 'THEAD' || theadtbody[j].tagName == 'TBODY') {
var tr=theadtbody[j].children;
for (var l = 0; l < tr.length; l++) {
if (tr[l].tagName == 'TR') {
th=tr[l].getElementsByTagName('th')[0];
th.parentNode.removeChild(th);
}
}
}
}
}
}
});
This JavaScript will change the appearance of the question in the survey, but it will not be apparent in the Editor. The JavaScript is only activated when the survey is being taken. Its effects can be seen in Preview Mode and when the survey is distributed. There are a couple more steps before we can preview it, however.
Step Six: Creating multiple EPA scales for multiple Concept Terms Now that we have all seven Evaluation, Potency, and Activity scales for the concept “Mother.” we can copy the group of them. To copy a group of questions, click on the first question, then hold down the SHIFT key, and click on all seven questions. As you click on additional questions, the Question Options Bar will disappear and be replaced by a smaller Multiple Selection Box, as shown in Figure . Select all seven questions.
Figure 13: Multiple Selection Box
After selecting these questions, click on “Copy” in the Multiple Selection Box. The copied block of questions will appear immediately below the original questions. Create as many copies of these questions as you have Concept Terms. After you create all the copies, add additional Concept Terms by replacing “Mother” with the appropriate term (Remember to use “HTML View”!).
Step Seven: “Blocking” Scales and Scale Order Randomization This tutorial covers the first three dimensions of randomization recommended by David Heise and described in the Introduction to this tutorial. As of this writing, I am still developing the random orientation of endpoints. The first dimension of randomization – random assignment to stimulus sets – is essentially a condition assignment issue. I will cover this briefly, as it is not an essential component to the actual creation of semantic differential scales. However, dimensions 2 and 3 have consequences for the construction of the scales so will be dealt with in more detail here.
At this point in the tutorial, I assume that you have a long list of questions created in Qualtrics. Each Concept Term has seven modified slider questions, with duplicate Concept Terms deleted, dimension-appropriate endpoints, and JavaScript in the questions containing the Concept Terms. In the following instructions, I will use FIVE Concept Terms, each with seven modified “slider” questions. You should have something like this:
Figure 14: Two (of Five) Concept Terms, with duplicate Concept Terms deleted, dimension-appropriate endpoints, and JavaScript in the questions containing the Concept Terms.
The next step is to break the questions into Blocks. Qualtrics can randomize questions at two levels. We will be using Block-level randomization (rather than Question-level), otherwise, each individual scale would be presented randomly, instead of each Concept Term. Right now, all the Concept Terms and questions are in a single Block. This is to make it faster and easier to edit questions. Now, however, we must separate each group of EPA Dimension Scales into its own block. Select Question 1 and 2, by clicking Question 1, holding the SHIFT key, and clicking Question 2.
Figure 15: Selecting Mother-Evaluation Dimensions to send to a New Block
In the Multiple Selection Box that pops up, click “Move to a new block”. Qualtrics will automatically scroll the screen to the end of the survey. Questions 1 and 2 will now be in a new Block. Rename the block so that you can identify it. Click “Block 2″ at the top left and enter a new name, such as “Mother E”
Figure 16: Renaming a Block
Now scroll back to the top of the survey, select the next two questions (i.e., Mother – Potency), select “Move to a new block”. Qualtrics will automatically scroll to the end of the survey and your new block. Rename the block (i.e., Mother P). Repeat this for all the EPA Dimension scales for all of the Concept Terms. After doing this, you will be left with the original block (Block 1). You can use this block to create demographic variables, or you can delete it by clicking on “Block Options” at the top right of the block, scroll to the bottom of the dropdown, click “Delete Block”, and confirm. At this point, you will have 3 blocks for each Concept Term. The next step is to randomize the Concept Terms and the scales. Scroll to the top of the Survey. In the Survey Options Bar (see Figure 1), click “Survey Flow”. A new window will pop up, showing all the blocks of your survey. Scroll to the bottom and select “Add a new Element Here.”
Figure 17: Survey Flow Screen
Step 7.1: In the box that pops up, select the pink button: “Randomizer”. This will add a pink Randomizer Box. Click on the button “Move” in the lower right of the box, and drag the Box to the top of the survey flow.
Figure 18: A Randomizer at the top of the survey flow.
Step 7.2: Click “Add a New Element Here” under the Randomizer Box. Add another Randomizer Box below this one.
Figure 19: A Randomizer “underneath” the first randomizer
Step 7.3: Now place the Blocks containing your first Concept Term under this second Randomizer by clicking “Move” in the lower right of the Block and dragging it:
Figure 20: Adding the three block for the Concept Term “Mother”.
Step 7.4: Now add another Randomizer Box, under the first box, by clicking “Add a New Element Here”4.
Figure 21: Adding a second Randomizer “underneath” the first. (Note the flow line, circled in red.)
Step 7.5: Add the next group of blocks containing your next Concept Term:
Figure 22: Second three-group block of Concept Term “Lawyer” added to Randomizer.
Step 7.6: Repeat steps 4 and 5, adding Randomizer Boxes and the rest of your Concept Term Blocks. When you finish, it should look like this:
Figure 23: Example of a properly formatted Survey Flow containing five Concept Terms
Step 7.7: Although blocks will be presented randomly, by default, Qualtrics does not prevent repeated presentation of questions. So, we need to make sure that respondents do not randomly see the same stimulus multiple times. This is easy to adjust. Make sure that the number of randomly presented elements is equal to the number of elements in the survey flow (Qualtrics will do this by default). Note that the first randomizer will have as many elements as you have concepts (five in this example), while the other randomizers will only have three elements.
In each of the pink Randomizer Boxes, select “Evenly Present Elements”. After you select it, an “Edit Count” button will appear next to it.
Figure 24: Evenly Presenting Elements in a Randomizer
Click “Edit Count”. A popup will appear. All the values will be set to zero by default. Change all of them to 1.
Figure 25: Preventing Random repetition of Randomized elements using Element Counts
Repeat this process for each Randomizer. Select “Evenly Present Elements”, and “Edit Count” to 1 for every element. Make sure you click “Save”. When you finish, click “Save Flow” at the bottom right of the Survey Flow Screen. Note that the first Randomizer (the one that presents the number of Concept Terms that you have) takes care of Heise’s second dimension of Randomization while each of the Randomizers containing three blocks for each Concept Term take care of the third dimension of randomization. To randomly present stimulus sets (the first dimension), you will need to add another Randomizer before the first Randomizer in the survey flow and place other conditions’ flows underneath it.
You are now ready to Preview and/or launch your survey.
Step 8: Appearance The final step is to use a survey template that is condusive to displaying the EPA slider scales. These templates can be found by clicking on “Look & Feel” among the Survey Options Tabs. Step 8.1: Click on “Look & Feel”. The Look & Feel window will pop up (see Figure 26).
Figure 26: Look & Feel window
Unfortunately, only three of the default Qualtrics templates display the EPA sliders properly. They are: “Business Modern Carbon (separated)” and “Qualtrics 3D” templates will display the Seven Scale Format properly. The “Qualtrics” template will display the Three Scale Format properly, but not the Seven Scale. The problem with the other templates is that they reduce the width of the Qualtrics screen display such that left-right scroll bars appear beneath each of the sliders (see Figure 27). In contrast, the above formats will display the sliders properly (see Figure 26 which uses “Qualtrics 3D”). Note: The only Duke University templates that display the sliders properly are “Duke Clinical Research Institute” and “Duke Maestro Care (It’s Alive) Wave 3″.
Figure 27: Qualtrics template that incorrectly displays left-right scroll bars beneath EPA sliders
Step 8.2: Select a template that displays the sliders properly. The lower part of the window shows a preview of the selected template. Do not choose a template that uses left-right scroll bars.
Step 8.3: Depending on your template, you may or may not want to check “Question Separator” at the top right of the Look & Feel window.
Troubleshooting
Problem: The scale labels do not extend to the ends of the slider scale (as in Figure 28). Solution: Make sure that “Center Labels” is selected for each question in the Question Options Bar.
Figure 28: Labels do not extend to the ends of the slider scale
Problem: There is a scroll bar underneath each slider or the sliders do not vertically line up with each other (as seen in Figures 29 and 30) Solution: You are using a template that does not display the EPA sliders properly (see Step 8: Appearance). Go to “Look & Feel” and select a different template.
Figure 29: Scroll bars under sliders (using Business Modern Forest).
Figure 30: Sliders do not line up properly (using default template: “Qualtrics”).
***Please send me any problems and I will add them to this section***
End Notes
1. David Heise (2010, pp.53-54) determined that 4.3 and -4.3 are (for measurement and statistical reasons) the best endpoints. In Surveyor, the appearance of the physical distance between the scale labels reflect this distance, where the distance between the labels Neutral, Slightly, Quite, and Extremely is equal to 1 and the distance between Extremely and Infinitely is equal to 1.3. As of this writing, I cannot change the appearance of the physical distance between Extremely and Infinitely on the slider scale, so the endpoint values must be 4,-4 in order to retain coherence between the Neutral – Extremely values. 2. On a standard QWERTY keyboard, the bar is above the ENTER key and shares its key with forward slash, \ . The reason we need to use the bar, is because we do not want to use the default “Grid Lines” in the Question Options Bar. This is because we will later add JavaScript to make the slider appear more like a Surveyor-style slider. The default grid lines present two problems: they superimpose number values on the slider scale (which we definitely do not want) and the line style is too large to fit properly on the Java-modified slider, distorting its appearance. 3. This includes any JavaScript, which is why we have not added it yet. Duplicate JavaScript on a single screen will distort the appearance of the question, rendering it unreadable on many devices. The proper way to enter JavaScript will be discussed in a moment. 4. You may use any of the “Add a New Element Here” buttons. If you create a new element (i.e., Randomizer) in the wrong place, you can just click “Move” and drag it to the correct position in the survey flow. ____________________________________________________________ Works Cited Heise, David R. 2010. Surveying Cultures. New Jersey: John Wiley & Sons, Inc. Here is an example of the above survey created using these instructions.