The Exam Development Process
My name is Star Grieser and I’m the Director of Testing for the Center for the Assessment of Sign Language Interpretation, CASLI. My goal here is to explain a bit about the exam development process, what’s involved, the time line, etc..
There has been some confusion over the difference between RID and CASLI. RID is the membership organization that, among many other things, awards national certification to Deaf and hearing interpreters who are able to meet the requirements for such. The RID certification process has several requirements, for example, a Bachelor’s or Associate’s degree, forty hours of training, etc., and those all fall under RID’s jurisdiction. One of those steps is to take and pass the exams, CASLI focuses solely on administering those exams and developing the news exams. RID set up CASLI as a separate LLC in 2016 to separate testing from their membership organization, and RID remains the primary stakeholder. There is an MOU between RID and CASLI where RID agrees to support CASLI until we are able to become fully autonomous.
The role of the Testing Committee is as an advisory capacity to CASLI. The Testing Committee is tasked with helping us develop and implementing the new exams. Also, as we administer and develop the exams, the Testing Committee can guide us on best practices in exam development, expert and volunteer recruitment, advise in hiring of consults., etc..
The Job Take Analysis, or JTA, is the first step in the whole process that begins as a study that determines the knowledge, skills, and abilities necessary for professionals in the field to be able to work successfully and is used to create the credential being studied. We gather a focus group of Subject Matter Experts (SMEs) and record the KSAs of the interpreting profession, and then with that information we send out a survey to the professionals in the field to verify the findings of the Focus Group. In 2015, RID conducted the NIC Job Task Analysis which is published here on our website. The Testing Committee felt there were sufficient differences between what hearing interpreters do and what Deaf interpreter do to warrant a separate (mini) Deaf Interpreter Job Task Analysis. From the results of the Deaf Interpreter Focus Group and JTA Survey, the Testing Committee will be able to reconcile the NIC JTA and the Deaf Interpreter JTA and use their findings to determine the design of the hearing interpreter exams and the Deaf interpreter exams.
Using the results of the Job Task Analysis study, the Testing Committee will determine what exam format will be used and outline the content of the exams. The Testing Committee is expected to convene over the summer to determine the exam designs and outlines, including the number and types of questions/scenarios for each exam.
Item writing is actually writing the content of the exams. An exam item is a question, scenario, a vignette, etc., we don’t know for sure yet as that is dependent on the Test Design. Volunteer Item Writers are provided training on how to write effective items and a bank of questions/scenarios is produced.
Each newly written item is reviewed for grammar, format, content, and relevance. Items that do not meet established standards are discarded or sent back to Item Writers for revision.
As we’re assessing a visual language, we expect invest quite a bit of time and resources on filming and editing of a portion of the exam content.
We will be either be reviewing contract with test delivery companies such as the ones we currently use, Castle Worldwide, LLC and Comira/PSI or may explore other companies, or explore building our own testing platform.
After the final draft of the test is loaded into our test delivery platform, we will conduct beta testing where the exam is opened to active candidates. Score reports for the initial set of exams are held pending Initial Scoring & Data Analysis. Once a critical number of exams have been taken, the scoring systems are reviewed to ensure that all questions and scoring systems are performing as expected.
We will convene a scoring committee who will determine the scoring criteria for the exams to ensure consistency, reliability and validity of the exams. Also, a cut score study will be conducted to set the passing point for the the exam.
Rater training will be developed and raters will be recruited to rate the performance parts of the exams. Raters will be trained and after training, assessed to see if they are consistently rating according the scoring criteria set but the scoring committee or if they are not consistent. Consistent raters will be invited to work as a rater for CASLI exams, while inconsistent raters may be retrained or be eliminated from the pool of rater applicants.
CASLI promises to release the exams for the hearing interpreters and for Deaf interpreters at the same time. One exam will not be released prior to another. This means our time line for exam development is pretty aggressive as all the exams for hearing interpreters and for deaf interpreters will have their own item writing committees, item reviewers, beta testers, scoring committees, etc.. This means we will need a lot of volunteers for the entire processes for both exams. Our goal is to be in, at least, the best testing phase by the end of 2019/beginning of 2020
Any questions or concerns? Contact me! Email Director@CASLI.org or call 703-988-4543. Also, check out our Testing Committee Call for Members page here.