Our previous blog post went into detail about the Item Review meeting for the Certified ScrumMaster (CSM
®) and Certified Scrum Product Owner (CSPO
®) tests. We suggest you read that post (and the others!) before continuing as it will create an informative background for this post.
The Standard Setting meeting is the last step in order to create a CSM test that is able to be published. What is a Standard Setting meeting? Great question! The simple answer is that the meeting will establish the pass or fail point (or cut score) for the test. The cut score is the minimum score that a potential certificant will need to obtain in order to pass the test. This is extremely important as we don’t want to set the cut score too high or else qualified people won’t pass the test. Similarly, we don’t want to set the cut score too low or else individuals that don’t understand Scrum will be able to pass it. Ultimately, we want to make sure the CSM certification continues to have value for our certificants!
So, how do we determine the cut score? As you’ve probably understood from our previous blog posts, it’s not a few Scrum Alliance staff sitting in a basement room deciding the cut score. It all starts with our community. We recruited 10 subject matter expert (SME) volunteers from the Scrum Alliance community who had varied backgrounds and experiences with Scrum. We also worked with Prometric, a leading test development company, to ensure that we’re following best practices.
Before the SMEs came to Denver though, we offered an opportunity to individuals that were eligible to take a CSM test. Individuals that still needed to take the CSM test were given two options when they logged into their dashboard: take the current test in its current format and receive the results immediately, or take a beta test where you won’t receive your results for about 6 weeks.
Individuals that chose the latter were incentivized by being entered into a drawing for a $300 USD Amazon or equivalent gift card. Individuals that passed the beta test would also earn the CSM certification for a period of four years instead of the standard two-year term. No one was required to take the beta test, but we were happy that close to 150 individuals took it!
The SMEs in Denver, with assistance from Prometric, then analyzed the data from the beta testers so they could see how real individuals scored on the test. The SMEs also took all four versions of the test – talk about a mind-numbing task! For each question on the test, the SMEs not only chose what they thought was the correct answer, but they also rated how likely a minimally qualified candidate would answer the question correctly. For example, on a more difficult question, they might mark that 40% of individuals will answer that question correctly. For an easier question, they might mark that 90% of individuals will answer that question correctly.
Prometric then gathers that information from all 10 SMEs and has the group discuss reasons why each volunteer marked a certain percentage. Based off the SME’s work history and experience, they might have different opinions of what a minimally qualified candidate would know and what a minimally qualified candidate would have trouble with. The SMEs then rescored their percentages based off the conversation. Prometric crunched the data and then provided Scrum Alliance with a recommendation of the cut score based off of the SMEs’ results.
Sounds like a busy two days, right? It was!
We couldn’t have accomplished so much without our wonderful volunteer SMES from the Scrum Alliance community! Thank you so much to Connie Fang, Luiz Quntela, Lynn Winterboer, Marcy DeMassa, Megan MacMillan, Michael Shelley, Ryan Knoblauch, Saif Ahmed, T. Anthony Basham, and Terrie Johnson.