CAA Licence Review

The next change occurred in 1988 when examinations became available on demand, rather than to a set timetable, which allowed candidates to sit the exam on completion of their study. This arrangement provided a higher level of service as some COs were prepared to conduct exams in the evenings or at weekends. A further improvement in service was provided in 1989 when COs were empowered to hold stocks of exam papers which reduced the time between requesting an exam and being able to sit it. A drawback to this arrangement was that frequent changes to aviation legislation soon rendered existing papers obsolete. Later that year, some COs were authorised to mark multi-choice examinations through the use of templates provided for the purpose.

The final major change occurred in 1992 with the introduction of the computerised Examfax system which utilised the now widespread facsimile system to transmit examination papers to COs, thus providing an even speedier service. Examfax randomly selects questions from a question library and is programmed to shuffle the answers to those questions to overcome the problem of candidates memorising the letter of the correct response. Unfortunately, there is clear evidence that some organisations, contrary to the specific requirement to destroy exam papers, have built up their own copies of the CAA exam library. Once the questions are known it is a relatively simple matter to determine the answers to those specific questions.

The Examfax system was generally acclaimed following its introduction; however, complaints began to emerge from concerned members of the industry that the system was being abused, and that candidates appeared to know the answers to questions without having a proper understanding of the broader issues involved. This, together with the known cases of fraud, as outlined above, prompted the CAA to initiate a review of the whole system. However, a review was timely in any case since the Aviation Licensing Branch, which was formed some three years ago, had combined flight crew and aircraft maintenance engineering licensing and aviation medicine together for the first time, the former two with their own examination systems. The Branch Head had determined that it was time to develop a common seamless system to cover all areas.

Late last year a small project team was established to carry out the review, their first task being to develop a questionnaire to be sent out to the 50,000 plus persons who receive the “Aviation Bulletin”. The questionnaire covered the major aspects of security, provision of service, scheduled exams versus on demand and, last but obviously not least, the questions of cost. Some several thousand replies were received which were most useful in the further development of a comprehensive discussion paper which canvassed the various options; this was then sent out to those who had indicated that they wished to participate in the further review process.

The project team is presently working on refining the Authority’s position on the issues involved, taking account of the further comments from industry, and it is expected that the industry will be informed of the final proposals by the end of the year. As outlined above, it is intended to introduce the new system on July 1 1995, which will give the industry at least six months to prepare for it. The options for any future examination system range between a return to essay type exams, rigidly controlled to set schedules, to multiple choice question papers available on a 24-hour basis, supervised by authorised persons outside the CAA.

The review team produced two schemes for consideration by industry which were termed System A and System B. System B was essentially a retention of the present system of multiple choice questions, with questions to be reused. However, exam papers would be produced in hard copy, not transmitted by fax. System A, the preferred system for aircraft maintenance engineers (AME) and flight crew (FC) examinations (except PPL and Air Law), may be summarised as follows:

Material to be examined to be classified into either factual recall material, e.g., the colour of Avgas, or material that requires the application of factual material to deduce solutions to engineering or operational situations. Factual recall material to be published in book form and tested by industry as convenient. The CAA to test deduced answers by essay type questions prepared by CAA staff and possibly accredited ground instructors. Essay questions not to be reused, permitting previous exam papers to be used as sample papers. The appointment of non-CAA persons as Conducting Officers be continued but with stricter controls and increased penalties for impropriety; e.g., in extreme cases loss of AOCs. Exam papers to be printed in hard copy in Canberra. CAA essay type examination candidates required to be recommended by an industry assessor. Applications to be made at least a week before the scheduled sitting and payment made at that time. CAA examinations to be held on a scheduled basis at CAA offices or approved industry venues. The frequency of sittings to be determined by the class of licence and demand. CAA essay examinations to be marked by technical specialists in the industry, using a standard marking proforma produced by the CAA. The markers to be appointed as delegates of the CAA, monitored and paid by the latter. Compulsory standardisation meetings of authorised markers to be held at least twice yearly.

The above recommendations do not apply to the PPL which the review team proposes should continue on the present non-scheduled basis using Examfax for the present. Cheating is evidently not a major problem with the PPL and the determination of the system best suited to applicants’ needs must, of necessity, await the outcome of the commercial examination recommendations. The project team has also recommended that Air Law exams be available on a non-scheduled basis to be sat at CAA District Offices. This recommendation stems mainly from the needs of overseas pilots seeking to operate in Australia.

There have been complaints that the new system of examinations for the ATPL introduced in April last year was not working. The failure rate has been excessively high, with in some instances 70 attempts having been made before a pass was obtained. The syllabus for the new system, which is based on the European system, was put out eighteen months before the first exams to give industry time to adjust. The Manager Aviation Licensing believes that the problem lies with the teaching organisations which are still teaching to meet the needs of the old system. He admits that there were problems following the introduction of the new system but that they have now been addressed.

One particular problem was the use of a generic model for the EFIS cockpit. It is recognised that this introduced more problems than it set out to resolve and it has now been decided to standardise on the 767 as the required model for study and examination purposes. In the past, there also existed a post-examination review panel consisting of representatives from the Association of Ground Training Organisations. This fell by the wayside following the introduction of the present system; however, it is now recognised that there is a need for such a panel to cover the industry as a whole, not just skewed towards the training industry, and it is thus proposed to establish it as part of the new system.

Trevor Burns, in concluding the briefing, said that there had been too much knee-jerk reaction in the past in an attempt to get a quick fix to a problem. He was determined that this time the job would be done properly and was satisfied that his team was tracking down the right path to produce the best result.

Comment

It is a fact of modern life that we are living in a society that sees the ends justifying the means. If you have to cheat to pass the exams that is OK, just don’t get caught. Unfortunately, in the case of commercial aviation, others’ lives are at stake and the CAA must take account of the “do what is comfortable for you” syndrome in setting up any new examination system. Unfortunately, the greater mass will be inconvenienced by tighter rules to overcome cheating, but that is the way it has to be.

It is also to be hoped that the industry has taken the time and effort to fully consider the issues involved and made a useful input of its views to the review team for the latter to take into account. Too often in the past there has been a tendency not to contribute in an effective manner, and then to snipe from the sidelines when the outcome is not to the individual’s liking. Finally, from my discussions with Trevor Burns, I reached the conclusion that the Aviation Licensing Branch is well aware of the problems and, with industry’s help, should produce a new system which balances the conflicting requirements of accessibility and security. I could not help reflecting, though, that a few more staff members might help speed the process along. Perhaps the Parliamentary enquiry could look into that one as well!

Additional Context

Although every effort is being made to increase the size of the CAA library, it is not presently large enough to overcome the problem. Interestingly, the FAA publishes the questions for US flight crew exams but it has a base of some 9000 questions. This means that there are some 30 questions on any given topic whereas the CAA presently only has five or six. A CAA Flying Operations Inspector who recently sat the US commercial licence exam has reported that it was a relatively easy matter to work through the questions and to recognise the answers without having to have any real understanding.

Technical Challenges In Question Writing

Although it might appear on face value that it should be a simple matter to build up the CAA question library, experience has shown that writing multiple choice questions requires a particular expertise. The Aviation Licensing Branch employs contractors to undertake the task but even the most experienced writer only achieves a 70% acceptability rate. The remainder are generally as low as 30%. The Branch tries to avoid having those who teach a particular subject employed as question writers as experience has shown that their success rate is not good. The problem of finding suitable writers is of sufficient magnitude that the Licensing Branch could not spend its budget last year.

Future Outlook For The Exam System

The need to change entirely the method of operating the CAA’s exam system has been obvious for some time. The new system, due to become operational mid 1995, should hopefully see the problems of the past being rectified in the most effective way possible. (Jim Thorn). Another problem is that vetting the questions to ensure that there is no ambiguity requires resources which the Licensing Branch physically lacks.