WEBVTT 00:00:06.090 --> 00:00:07.820 position:50% align:middle Hi, everyone. 00:00:07.820 --> 00:00:12.100 position:50% align:middle Welcome to the REx-PN Conference, and thank you for joining this session. 00:00:12.100 --> 00:00:15.620 position:50% align:middle My name is Shu-chuan Koa. 00:00:15.620 --> 00:00:20.940 position:50% align:middle I'm the senior manager of measurement and testing at NCSBN. 00:00:20.940 --> 00:00:27.880 position:50% align:middle Today, we are going to cover the Computerized Adaptive Testing, also known as CAT. 00:00:27.880 --> 00:00:34.213 position:50% align:middle In today's presentation, you will understand the concept of Computerized Adaptive Testing 00:00:34.213 --> 00:00:38.538 position:50% align:middle and its three important components. 00:00:38.538 --> 00:00:41.108 position:50% align:middle What is adaptive testing? 00:00:41.108 --> 00:00:45.527 position:50% align:middle Adaptive testing is originally called tailored testing. 00:00:45.527 --> 00:00:52.097 position:50% align:middle It is designed to adjust the level of difficulty to match your ability as a test taker. 00:00:53.548 --> 00:00:58.318 position:50% align:middle Every exam is customized for the person taking it. 00:00:58.318 --> 00:01:00.528 position:50% align:middle How does it work? 00:01:00.528 --> 00:01:06.358 position:50% align:middle The computer algorithms select items from a large item bank for you to answer. 00:01:06.358 --> 00:01:13.278 position:50% align:middle The computer aims to identify your ability and give you items that are challenging to you. 00:01:13.278 --> 00:01:19.618 position:50% align:middle If you are a high-ability candidate, you will receive more difficult items because these 00:01:19.618 --> 00:01:24.388 position:50% align:middle items can identify how high your ability is. 00:01:24.388 --> 00:01:29.008 position:50% align:middle If your ability is low, you will receive more easy items that you 00:01:29.008 --> 00:01:31.618 position:50% align:middle can possibly answer. 00:01:31.618 --> 00:01:38.638 position:50% align:middle The purpose of the CAT exam is not to tell the percentage of items you can answer correctly. 00:01:38.638 --> 00:01:46.208 position:50% align:middle The REx-PN CAT exam is a different type of exam that estimates your ability in real-time and compares your 00:01:46.208 --> 00:01:50.268 position:50% align:middle ability to the passing criteria. 00:01:50.268 --> 00:01:55.898 position:50% align:middle Every time when you answer an item, the computer estimates your ability based on all 00:01:55.898 --> 00:02:01.075 position:50% align:middle questions you have answered, and it then select an item that is comparable 00:02:01.075 --> 00:02:03.485 position:50% align:middle to your ability estimate. 00:02:03.995 --> 00:02:09.405 position:50% align:middle If you answer an item correctly, the computer will find a more difficult item 00:02:09.405 --> 00:02:11.625 position:50% align:middle for you to answer. 00:02:11.625 --> 00:02:19.285 position:50% align:middle Will the computer just randomly select any items that is more difficult than the item you just responded to? 00:02:19.285 --> 00:02:20.615 position:50% align:middle Nope. 00:02:20.615 --> 00:02:26.495 position:50% align:middle We program the computer in a way that the next item selected for you will give you a 50% chance 00:02:26.495 --> 00:02:29.415 position:50% align:middle to answer it correctly. 00:02:29.415 --> 00:02:33.827 position:50% align:middle When you see this item, you may feel that you can do it, 00:02:33.827 --> 00:02:38.987 position:50% align:middle but you are not 100% sure your answer will be correct. 00:02:38.987 --> 00:02:45.667 position:50% align:middle Based on the statistical models, the 50% probability of answering an item correctly can 00:02:45.667 --> 00:02:49.937 position:50% align:middle give us the most information about your ability. 00:02:49.937 --> 00:02:54.737 position:50% align:middle Don't worry, I'm not going to bore you with statistics today. 00:02:54.737 --> 00:03:02.490 position:50% align:middle Instead, I want to emphasize to you that we have a large enough item bank to support the CAT exam. 00:03:02.490 --> 00:03:08.420 position:50% align:middle If your ability is very high, we have very difficult items for you. 00:03:08.420 --> 00:03:14.670 position:50% align:middle If your ability is very low, we still can find very easy items for you. 00:03:14.670 --> 00:03:17.580 position:50% align:middle The CAT exams are challenging for all test takers. 00:03:17.580 --> 00:03:24.790 position:50% align:middle If you take a CAT exam and start to see difficult items, that means your ability estimate is 00:03:24.790 --> 00:03:29.800 position:50% align:middle getting high, and you may have a good chance to pass the exam. 00:03:29.800 --> 00:03:35.203 position:50% align:middle Every time you respond to an item, the computer re-estimates your ability 00:03:35.203 --> 00:03:37.896 position:50% align:middle based on all questions you have answered. 00:03:37.896 --> 00:03:45.784 position:50% align:middle And according to the updated ability estimate, the computer finds you the matching item for you to answer. 00:03:45.800 --> 00:03:51.346 position:50% align:middle The more items you answer, the more the computer knows about your ability. 00:03:51.346 --> 00:03:57.906 position:50% align:middle Therefore, the computer is able to give you a more precise ability estimate. 00:03:57.906 --> 00:04:02.550 position:50% align:middle This is a graphical representation of the adaptive algorithms. 00:04:02.550 --> 00:04:08.740 position:50% align:middle The y-axis is the scale used for both item difficulty and the candidate ability. 00:04:08.740 --> 00:04:15.280 position:50% align:middle On this scale, a larger number always indicates a harder question and higher ability. 00:04:15.280 --> 00:04:21.650 position:50% align:middle A smaller number always indicates an easier question and a lower ability. 00:04:21.650 --> 00:04:25.940 position:50% align:middle The x-axis is simply a tracker of the number of operational questions. 00:04:25.940 --> 00:04:30.452 position:50% align:middle The horizontal blue line is the passing standard. 00:04:30.452 --> 00:04:38.202 position:50% align:middle This is the criteria that a computer uses to determine if a candidate should pass or fail the exam. 00:04:38.202 --> 00:04:45.052 position:50% align:middle It reflects the ability level of a minimally competent candidate on the REx-PN exam. 00:04:45.052 --> 00:04:49.782 position:50% align:middle If you are interested in knowing how the passing standard is determined, 00:04:49.782 --> 00:04:52.902 position:50% align:middle please attend the presentation for standard setting. 00:04:52.902 --> 00:05:00.480 position:50% align:middle To begin an exam, the CAT algorithms will select an item that is easier than the passing standard. 00:05:00.480 --> 00:05:05.300 position:50% align:middle The blue dot represents the item difficulty of the item selected. 00:05:05.300 --> 00:05:10.930 position:50% align:middle When the first question is answered, the computer starts to identify the candidate's ability, 00:05:10.930 --> 00:05:15.790 position:50% align:middle which is represented by the black square. 00:05:15.790 --> 00:05:21.870 position:50% align:middle The line above and below the black square is the 95% confidence interval. 00:05:21.870 --> 00:05:28.900 position:50% align:middle This is the mathematical range within which the computer thinks the candidate's true ability is. 00:05:28.900 --> 00:05:35.413 position:50% align:middle Since the first question is answered correctly, the computer starts to estimate the candidate's ability 00:05:35.413 --> 00:05:42.393 position:50% align:middle and then selects a more difficult item to administer, and that the item difficulty of the second item is very 00:05:42.393 --> 00:05:46.273 position:50% align:middle close to the updated ability estimate. 00:05:46.273 --> 00:05:51.933 position:50% align:middle After the candidate answers the second question, the computer evaluates how well the candidate has 00:05:51.933 --> 00:05:55.163 position:50% align:middle performed on the first two questions. 00:05:55.163 --> 00:06:01.107 position:50% align:middle Since the first two questions are answered correctly, the computer gives a higher ability estimate and then 00:06:01.107 --> 00:06:04.507 position:50% align:middle selects a more difficult item to administer. 00:06:04.507 --> 00:06:11.187 position:50% align:middle Again, this item has the item difficulty very close to the updated ability estimate. 00:06:11.187 --> 00:06:17.267 position:50% align:middle When an item is inserted incorrectly, the ability estimate will get lower than the previous 00:06:17.267 --> 00:06:24.567 position:50% align:middle one and an easier item that can match the updated ability estimate is selected. 00:06:24.567 --> 00:06:28.363 position:50% align:middle This adaptive process will go on as the exam continues. 00:06:30.357 --> 00:06:37.102 position:50% align:middle Now, I would like to introduce you to the three components of the REx-PN CAT. 00:06:37.102 --> 00:06:42.782 position:50% align:middle In the REx-PN exam, all candidates have four hours to answer questions. 00:06:42.782 --> 00:06:47.652 position:50% align:middle The length of the exam will vary depending on the candidate's ability. 00:06:47.652 --> 00:06:53.665 position:50% align:middle The minimum nurse exam will have 90 items, which comprise of 60 operational items 00:06:53.665 --> 00:06:56.332 position:50% align:middle and the 30 pretest items. 00:06:56.332 --> 00:07:01.786 position:50% align:middle The operational items have the known item difficulties obtained from the pretest, 00:07:01.786 --> 00:07:07.408 position:50% align:middle so the CAT algorithms can use these items to estimate your ability. 00:07:07.408 --> 00:07:10.248 position:50% align:middle The pretest items are new items. 00:07:10.248 --> 00:07:17.858 position:50% align:middle We'll use your responses and ability estimate to determine item difficulties of the new items. 00:07:17.858 --> 00:07:24.708 position:50% align:middle The maximum length exam will have 120 operational items and the 30 pretest items. 00:07:24.708 --> 00:07:30.163 position:50% align:middle The exam can stop with minimum number of questions if your ability is far above 00:07:30.163 --> 00:07:32.161 position:50% align:middle or below the passing standard. 00:07:33.066 --> 00:07:39.786 position:50% align:middle This means when you respond to 60 operational items, the computer knows that you will definitely pass or 00:07:39.786 --> 00:07:45.976 position:50% align:middle fail the exam and that there is no need for you to respond to more questions. 00:07:45.976 --> 00:07:52.776 position:50% align:middle If your ability is very close to the passing standard, you will be given a longer exam because the computer 00:07:52.776 --> 00:07:57.766 position:50% align:middle needs to collect more information about your ability to make the pass/fail decision. 00:07:59.646 --> 00:08:05.128 position:50% align:middle On this slide, I would like to introduce you to content balancing. 00:08:05.128 --> 00:08:12.768 position:50% align:middle Content balancing means that all candidate receives the same distribution of items by content area. 00:08:12.768 --> 00:08:18.678 position:50% align:middle For the REx-PN exam, we use Test Plan as a blueprint of all exams. 00:08:18.678 --> 00:08:25.548 position:50% align:middle The Test Plan provides the summary of the content and the scope of the licensure examination. 00:08:25.548 --> 00:08:31.728 position:50% align:middle No matter how many items you have in your exam, your exam meets the specified percentages 00:08:31.728 --> 00:08:32.885 position:50% align:middle of the Test Plan. 00:08:34.528 --> 00:08:41.928 position:50% align:middle Oh, you may wonder how we can assure the content coverage is the same for all CAT exams. 00:08:41.928 --> 00:08:45.218 position:50% align:middle We programmed the computer to do two things. 00:08:45.218 --> 00:08:51.998 position:50% align:middle Firstly, before selecting the next question, the computer will determine the percentage of items you 00:08:51.998 --> 00:08:58.908 position:50% align:middle currently have in all A content areas and to find out which content area deviates the most 00:08:58.908 --> 00:09:00.231 position:50% align:middle from the Test Plan. 00:09:01.557 --> 00:09:09.097 position:50% align:middle Secondly, the computer will select an item that targets to your ability from that content area. 00:09:09.097 --> 00:09:15.867 position:50% align:middle This way, no matter how many items you have in your exam, your exam can comply with the Test Plan. 00:09:17.547 --> 00:09:22.187 position:50% align:middle Starting from this slide, I'm going to show you how the computer decides whether 00:09:22.187 --> 00:09:26.237 position:50% align:middle you should pass the REx-PN exam. 00:09:26.237 --> 00:09:31.594 position:50% align:middle In the REx-PN exam, we only report the pass/fail classification. 00:09:31.594 --> 00:09:38.614 position:50% align:middle You have four hours to take the exam, and you have to respond to at least 60 operational items. 00:09:38.614 --> 00:09:44.074 position:50% align:middle Once you have answered 60 operational items, your ability estimate is compared 00:09:44.074 --> 00:09:46.234 position:50% align:middle to the passing standard. 00:09:46.234 --> 00:09:53.654 position:50% align:middle The computer has three different rules to terminate the exam and makes the pass/fail decision. 00:09:53.654 --> 00:09:58.994 position:50% align:middle The first stopping rule is the 95% confidence interval rule. 00:09:58.994 --> 00:10:05.924 position:50% align:middle After you have answered 60 operational items, the computer starts to check if your ability is clearly 00:10:05.924 --> 00:10:07.754 position:50% align:middle above or below the passing standard. 00:10:09.047 --> 00:10:14.043 position:50% align:middle The exam continues when the confidence interval straddles the passing standard. 00:10:14.184 --> 00:10:20.364 position:50% align:middle The exam stops when the confidence interval is clearly above or below the passing standard. 00:10:20.364 --> 00:10:24.072 position:50% align:middle When the confidence interval of your ability is above the passing standard, 00:10:24.072 --> 00:10:27.048 position:50% align:middle you pass the exam, congratulations. 00:10:27.048 --> 00:10:33.809 position:50% align:middle When the confidence interval is below the passing standard, then the person failed the exam. 00:10:33.809 --> 00:10:38.529 position:50% align:middle This is a graphical representation of a minimum length exam. 00:10:38.529 --> 00:10:43.279 position:50% align:middle Again, the blue horizontal line represents the passing standard. 00:10:43.279 --> 00:10:50.220 position:50% align:middle The test stops at item 60 because it is the minimum number of operational questions required. 00:10:50.839 --> 00:10:57.889 position:50% align:middle And at that point, the final ability and the confidence interval are below the passing standard. 00:10:57.889 --> 00:11:03.762 position:50% align:middle In this case, the computer concludes that the candidate failed the exam. 00:11:03.762 --> 00:11:06.272 position:50% align:middle This slide will show you a different scenario. 00:11:06.272 --> 00:11:13.342 position:50% align:middle When you answer 60 operational items, the computer cannot decide whether or not you should 00:11:13.342 --> 00:11:18.592 position:50% align:middle pass the exam and continues to select items for you to answer. 00:11:18.592 --> 00:11:27.492 position:50% align:middle The exam stops at item 105, because after this item, your ability estimate and confidence interval are 00:11:27.492 --> 00:11:30.462 position:50% align:middle above the passing standard. 00:11:30.462 --> 00:11:35.112 position:50% align:middle Yeah, congratulations, you pass the REx-PN exam. 00:11:35.112 --> 00:11:39.802 position:50% align:middle Now let's look at a second rule to terminate a CAT exam. 00:11:39.802 --> 00:11:49.202 position:50% align:middle If your ability is very close to the passing standard, this stopping rule may be used to terminate your exam. 00:11:49.202 --> 00:11:56.242 position:50% align:middle When you answer 120 operational items, the computer disregards the 95% certainty requirement 00:11:56.242 --> 00:12:01.760 position:50% align:middle and then makes the pass/fail decision based on your final ability estimate, 00:12:01.760 --> 00:12:07.630 position:50% align:middle which is calculated based on all 120 questions you have answered. 00:12:07.630 --> 00:12:13.230 position:50% align:middle If your final ability is above the passing standard, you pass the exam. 00:12:13.230 --> 00:12:21.040 position:50% align:middle If your final ability is at or below the passing standard, sorry, you fail the exam. 00:12:21.040 --> 00:12:25.510 position:50% align:middle This example will illustrate a maximum length exam. 00:12:25.510 --> 00:12:31.072 position:50% align:middle The computer concludes that you pass the exam because the final ability is above the passing standard, 00:12:31.072 --> 00:12:36.912 position:50% align:middle even though the confidence interval is not totally above the passing standard. 00:12:36.912 --> 00:12:44.232 position:50% align:middle If you use all four hours exam time then your exam has not been terminated by either the 95% Confidence 00:12:44.232 --> 00:12:47.732 position:50% align:middle Interval rule or the maximum length rule. 00:12:47.732 --> 00:12:52.782 position:50% align:middle The running out of time rule will be applied to stop your exam. 00:12:52.782 --> 00:12:57.862 position:50% align:middle This means when the exam time is up, the computer again disregards the 95% certainty 00:12:57.862 --> 00:13:04.448 position:50% align:middle requirement and then uses your final ability to make the pass/fail decision. 00:13:04.448 --> 00:13:10.208 position:50% align:middle Again, your final ability is calculated based on all questions you have answered. 00:13:10.208 --> 00:13:16.938 position:50% align:middle If you have answered at least 60 operational items and then your final ability is above the passing standard, 00:13:16.938 --> 00:13:19.138 position:50% align:middle you pass the exam. 00:13:19.138 --> 00:13:25.758 position:50% align:middle If you have not answered at least 60 operational items, or your final ability is at or below the 00:13:25.758 --> 00:13:30.068 position:50% align:middle passing standard, you fail the exam. 00:13:30.068 --> 00:13:33.206 position:50% align:middle This example will demonstrate an exam that ran out of time. 00:13:33.206 --> 00:13:39.306 position:50% align:middle When you answer 60 operational questions, the computer starts to compare your ability 00:13:39.306 --> 00:13:41.316 position:50% align:middle to the passing standard. 00:13:41.316 --> 00:13:44.966 position:50% align:middle When the exam time is up, the exam stops. 00:13:44.966 --> 00:13:51.543 position:50% align:middle The computer concludes that you passed the exam because the final ability is above the passing standard. 00:13:52.506 --> 00:13:54.866 position:50% align:middle This is another example. 00:13:54.866 --> 00:13:58.826 position:50% align:middle Please note that this candidate's ability is very high. 00:13:58.826 --> 00:14:06.105 position:50% align:middle This candidate failed the exam because he or she does not take enough operational items within four hours. 00:14:06.105 --> 00:14:10.505 position:50% align:middle This is the last example of today's presentation. 00:14:10.505 --> 00:14:13.805 position:50% align:middle When the exam time is up, the exam stops. 00:14:13.805 --> 00:14:17.535 position:50% align:middle The computer evaluates the final ability estimate. 00:14:17.535 --> 00:14:22.915 position:50% align:middle This candidate failed the exam because the final ability is below the passing standard. 00:14:22.915 --> 00:14:25.625 position:50% align:middle We have covered a lot of information today. 00:14:25.625 --> 00:14:31.284 position:50% align:middle If you want to know more about REx-PN CAT, please visit rexpn.com. 00:14:32.569 --> 00:14:38.314 position:50% align:middle Please enter any questions you might have in the Q&A box, and then we will answer them in the 00:14:38.314 --> 00:14:39.804 position:50% align:middle next Q&A session. 00:14:39.804 --> 00:14:42.816 position:50% align:middle Thank you for your time.