The 2010 WoW Learning Survey’s Design

General:

I spent a substantial amount of time designing the first survey for the WoW Learning project. I was initially unsure what data would be useful, but I knew I wanted enough data to make statements about particular groups of people: men were more social, most women created tank characters initially, millennials were using WoW more for learning, etc. That resulted in the three-part design of the survey: in-game demographic data, the essay question about play motivations, and real-world demographic details. Privacy was important and encouraging people to complete was also important. It was reasoned that in-game demographics were details people would not be as sensitive about, so they were asked for first and real-world demographic details left until the end. It was also recognized that many people would not complete the essay section. Putting the in-game demographic details first meant that certain types of data could be collected that could also be used for other purposes, e.g. answering whether women initially choose healer characters. After use by a small test audience, the survey was modified to include sample answers or explanations of how answers should be calculated.

Where possible and sensible, permissible option lists were used to help reduce the need for data standardization after the survey. For example, I know there are only so many WoW character classes and roles. I know that players can only belong to one of several types of realms. In the case where I wanted time estimates from players, providing a list of ranges means everyone’s has the same degree of accuracy and is expressed in the same units. This was not deemed necessary for year of birth, country of residence, and nationality; they were left as free text. The birth year worked out fine, but some normalization had to be done on the countries and nationalities, e.g. English and Scottish changed to British and Belgium changed to Belgian, etc.

Read more to download the survey as a PDF and see options for specific questions.

Details on Specific Options:

The attached PDF shows the design of the survey as it appeared on SurveyMonkey, the third-party commercial survey hosting service I used. However, because it’s been saved as a PDF from a web browser, you can’t see the drop-down menu items. Where that happens, I’ve described the questions and their options in this section.

Part 1: Q1: How long have you been playing WoW (to the nearest half year)? If you played in the public or closed beta, you can count that too.

Options here were given to the nearest half year, up to 6.5 years:

0.5 years
1.0 year
1.5 years
2.0 years
2.5 years
3.0 years
3.5 years
4.0 years
4.5 years
5.0 years
5.5 years
6.0 years
6.5 years

At the time of the survey, it wasn’t possible to have played more than 6.5 years.

Part 1: Q2: Tell us about the first character you created. Tell us about the character on which you currently spend most of your time. Tell us about the character you enjoy playing the most.

This was broken down into race, class, role, and time played for each of the questions. For race:

Blood Elf
Draenei
Dwarf
Gnome
Human
Night Elf
Orc
Tauren
Troll
Undead

For class:

Death Knight
Druid
Hunter
Mage
Paladin
Priest
Rogue
Shaman
Warlock
Warrior

For role:

Healer
Tank
Ranged DPS
Melee DPS
Pure Role-Playing

Pure role-playing was added for people who were using the game as a world in which to base their storytelling rather than engaging with the provided gaming content elements. With the population I was recruiting primarily from, I did not expect there to be many role-players and the option was missing from my original design, but I added it after testing.

For time played, options were given to the nearest 15 days until 300 days. After that, it jumped to the nearest 100 days. It’s important to note that the time asked for here is /played time. That is a measure of time spent actually in the game and not a measure of calendar time. This information is available for any WoW character you can still play. I ask for /played because I am trying to assess a person’s game experience. You could have two players who have both played the game for 6 calendar months, but one might have spent 20 hours a week versus the other only 1 hour. See the discussion in /Played Time as a Measure of WoW Experience. Possibilities here then were:

Less than 15 days
15
30
45
60
75
90
105
120
135
150
165
180
195
210
225
240
255
270
285
300
More than 300 days
More than 400 days
More than 500 days
More than 600 days
More than 700 days
More than 800 days
More than 900 days
More than 1000 days
No idea

Part 1: Q3: The next choices relate to the type of game realm where you play. If you play on more than one, pick the realm used by the character on which you spend the most time.

This was broken down into type of realm and location. For realm type:

PvE (Player versus Environment)
PvP (Player versus Player)
RP (Role-Play)
RP-PVP (Role-Play Player versus Player)

For realm location:

China
Korea
Europe
US

Part 1: Q5: How much time per week in hours do you estimate that you spend on average in game? If it’s more than 40, choose that from the menu and enter the number of hours into the short comment field provided.

This was broken down into 2-hour chunks:

Less than 1 hour
1 to 2 hours
2 to 4 hours
4 to 6 hours
6 to 8 hours
8 to 10 hours
10 to 12 hours
12 to 14 hours
14 to 16 hours
16 to 18 hours
18 to 20 hours
20 to 22 hours
22 to 24 hours
24 to 26 hours
26 to 28 hours
28 to 30 hours
30 to 32 hours
32 to 34 hours
34 to 36 hours
36 to 38 hours
38 to 40 hours
More than 40 hours

One survey design flaw I see often is, when providing a list of options, not covering edge cases where it is not applicable at all or someone does something you did not expect. Although I would expect that all participants have played the game and are regularly playing the game, there might be some who do not play at least an hour per week or who are spending more than a full-time job in the game.

Last Bit: Q1: If you’re happy to be contacted with any queries about your responses for this survey or to participate in future surveys, you can leave an e-mail address. This will be associated with your survey responses.

This was an optional question as I was purposefully trying to avoid collecting any identifiable information about participants. However, I could foresee circumstances where it might be useful to follow up on responses or issue invitations to future research surveys. They had simple yes/no answers, with the default being “No”.