CHAPTER 2
Proctor Instructions for Administering the Examination

The PROPS Test is administered in four consecutive parts. Each part and question are generally aligned with the common stages associated with the preplanning and execution of a UAS mission. The proctor should read each question to the RPIC orally and assess their understanding of the topics and overall proficiency. For each question, there is a minimum “performance/knowledge” level and “evaluation guidance.” The minimum performance/knowledge level indicates the minimum performance or level of knowledge required for a Level 2 or 3 certification. The examination questions and certification are specific to a particular aircraft. The RPIC should have access to the aircraft for the examination. Additionally, the RPIC should have access to all of the state DOTʼs tools and resources to address the question. This includes the manufacturer manuals, software, online applications, checklists and DOT-specific documents. The RPIC may not use other UAS operators for assistance during the exam.

Part 1 of the examination focuses on pre-mission activities. It can be administered inside, where the RPIC has access to hard copy or online resources. Part 2 of the examination assesses the RPICʼs familiarity with a specific aircraft. The RPIC will need to have access to the aircraft and controller to complete these questions. The aircraft and controller will need to be energized; however, the questions can be answered inside before relocation to the mission site. Part 3 of the examination is a series of untimed flight activities at the mission site. The RPIC will be responsible for operating the UAS and performing tasks detailed by the proctor. Part 4 of the examination is a timed assessment verifying the RPICʼs stick skills. This examination has adopted the NIST BPERP flight assessment for this portion of the assessment. Parts 3 and 4 require the use of the NIST Open Test Lane. Fabrication instructions for the Open Test Lane provided by NIST and the obstruction cubes are provided in Appendices A and B, respectively.

Mission Site and Environmental Conditions

Parts 3 and 4 of the examination require the RPIC to conduct various UAS flight maneuvers. Both Parts 3 and 4 utilize the NIST Open Test Lane with an S distance (the distance between stands) of 10 feet. In total, the NIST Open Test Lane has a minimum footprint of 100 feet long by 40 feet wide. However, it is recommended that the examination be performed on a site where the UAS can be flown over a 5-acre area without concerns about impacting ground activity by flying over bystanders.

For consistency purposes, it is recommended that the examination be administered during daytime hours, with low wind (<5 miles per hour) and high visibility [>10 statute miles (SM)]. However, at the state DOTʼs discretion or RPICʼs consent, these environmental conditions can be waived for convenience.

Assessment Criteria

The criteria for the PROPS Test are drawn from ASTM F3266, Standard Guide for Training for Remote Pilot in Command of Unmanned Aircraft Systems (UAS) Endorsement. This standard defines the knowledge, skills, and abilities necessary for safe commercial UAS operations. The most relevant sections are Section 5, General Knowledge Subject Matter Requirements, and Section 6, Indoctrination Curriculum Requirements. Section 5 notes that “general knowledge” can be tested through written examinations, such as the FAA Unmanned Aircraft General (UAG) exam. For this reason, the PROPS Test emphasizes evaluating proficiency in the areas outlined in Section 6.

ASTM F3266 organizes proficiency into “tasks” and “knowledge.” Task proficiency requires both task performance and task knowledge. A four-tier matrix from ASTM F3266 (Table 1), is included in the PROPS Test to guide proctors in assessing flight proficiency. Task performance is scored on a scale of 1 (limited) to 4 (proficient), while task knowledge ranges from “a” (knows nomenclature) to “d” (knows advanced theory). Both are measured together. Subject knowledge is evaluated separately on a four-tier scale, from “A” (knows facts) to “D” (knows evaluation).

ASTM F3266 establishes minimum proficiency levels using this four-tier matrix. This assessment includes questions on each flight proficiency topic from the standard and provides state DOT proctors with the tools and context needed to determine whether an RPIC meets the standardʼs minimum proficiency criteria.

Source: ASTM F3266.

Long Description.

A table shows the ASTM F3266 task and knowledge levels. The table has 2 columns and 16 rows. The column headings are “Scale Value” and “Definition.” The first section of the table lists Task Performance Levels, of which there are four: Task Performance Level 1, Is limited (Can do simple parts of the task. Needs to be told or shown how to do most of the task.); Task Performance Level 2, Is partially proficient (Can do most parts of the task. Needs only help on hardest parts.); Task Performance Level 3, Is competent (Can do all parts of the task. Needs only a spot check of completed work.); and Task Performance Level 4, Is proficient (Can do the complete task quickly and accurately. Can tell or show others how to do the task.). The second section of the table lists Task Knowledge Levels, of which there are four, labeled “a” to “d”: Task Knowledge Level a, Knows nomenclature (Can name parts, tools, and simple facts about the task.); Task Knowledge Level b, Knows procedures (Can determine step by step procedures for doing the task.); Task Knowledge Level c, Knows operating principles (Can identify why and when the task must be done and why each step is needed.); and Task Knowledge Level d, Knows advanced theory (Can predict, isolate, and resolve problems about the task.). The third section of the table lists Subject Knowledge Levels, of which there are four, labeled “A” to “D”: Subject Knowledge Level A, Knows facts (Can identify basic facts and terms about the subject.); Subject Knowledge Level B, Knows principle. (Can identify relationship of basic facts and state general principles about the subject.); Subject Knowledge Level C, Knows analysis (Can analyze facts and principles and draw conclusions about the subject.); and Subject Knowledge Level D, Knows evaluation (Can evaluate conditions and make proper decisions about the subject.).

Question Evaluation

Parts 1, 2, and 3 of the PROPS Test consist of open-ended questions asked orally by the exam proctor to the RPIC. Each question is tied to at least one section of ASTM F3266, which defines the baseline knowledge or skill needed to show competency. This required standard is shown in Table 1.

Task evaluations use two measures: task performance, scored with a number, and task knowledge, scored with a lowercase letter. Subject knowledge is evaluated separately on a single standard, shown with an uppercase letter. Exam proctors compare the RPICʼs responses against the assessment criteria provided with each question, which are mapped to the ASTM task and knowledge level matrix. Each question includes a scoring table, which must be used to record the evaluation. The bolded cell in the table marks the minimum required level of understanding. In the example in Figure 1, the proctor assessed that the RPIC “Is competent” and “Knows operating principles” of the task. This meets the requirement for the task performance and exceeds the requirement for the task knowledge.

Clarifications, Assumptions, and Limitations

Each of the questions indicates which ASTM section it addresses. Some of the ASTM F3266 sections have different levels of performance and knowledge levels. In this situation, the examination assesses all ASTM F3266 sections associated with the question at the highest of the levels.

Many of the aspects of flight proficiency are addressed through standardized knowledge testing. The ASTM F3266 standard acknowledges FAAʼs Unmanned Aircraft General (UAG) examination as an acceptable means for meeting the general knowledge requirement of flight proficiency. The ASTM F3266 sections in Appendix C are assumed to be assessed with the UAG exam.

The ASTM F3266 standard addresses various types of UAS frames and fuel sources. For example, some of the sections address fixed-wing or gas-powered aircraft. The PROPS Test is intended for off-the-shelf, multirotor, electric-powered UAS. The ASTM F3266 sections in Appendix D were not appropriate for this type of aircraft and were not included in the assessment.

Two ASTM F3266 standards were not included in the PROPS Test, as their assessment would place an excessive burden on the state DOTs. Section 6.4.1.3 addresses wind and “the effects on ground track and speed with and without GNSS.” Many off-the-shelf UASs available to state DOTs do not have the ability to manually deactivate the Global Navigation Satellite System/Global Positioning System (GNSS/GPS) stabilizing feature.

Additionally, simulating wind, especially in confined space, would require an overly sophisticated and costly apparatus to assess. Similarly, Section 6.4.1.5 addresses flight mode, specifically, “the recognition of all likely flight mode failures and the appropriate recovery from or mitigation of those failures.” Recognizing flight mode failures and identifying appropriate recovery strategies are addressed in the examination. However, simulating the failure is not easily performed with many common off-the-self UASs.

Long Description.

Image of a table used to score the PROPS test. The table has 5 columns and 2 rows. Row 1: Task Performance; 1; 2; 3; 4. Row 2: Task Knowledge; a; b; c; d. The cells with 3 and c have green check marks that indicate the level the sample pilot earned. Cell 3 in the task performance row and cell b in the task knowledge row are emphasized to indicate the minimum required level of understanding.