Knowledge Base
Occupational Test Administration and Use
Eatwell, J and Direen, K (1997). Occupational Test Administration and Use. In H. Love & W. Whittaker (Eds.) Practice issues for clinical and applied psychologists in New Zealand. Wellington: The New Zealand Psychological Society.
Psychometric assessments are used to enhance the quality and quantity
of information available for selection, development and training decisions
and as an aid to organisational change. The highest standards of practice
in the use of all psychometric tools are needed in order to maximise the
benefit to organisations and the individuals assessed, and to promote
fairness and equality of opportunity for all.
To facilitate this the following guidelines have been produced for practitioners
utilising psychometric assessment instruments. The guidelines aim to cover
when, and who should use, psychometric tools, the principles in choosing
appropriate tools, the preparation of candidate and administration of
tests, and the issues surrounding confidentiality and storage of materials
and results.
When Should Assessment Instruments Be Used?
Where competencies have been identified as required for effective job
performance, psychometric assessments may be used to measure these for
selection either for short listing candidates or final selection decision
making, placement or promotional decisions, development, team building,
counselling, outplacement and organisational development. In each case
the situation must be evaluated to decide whether the use of assessment
instruments would be appropriate to help achieve the desired human resource
objectives.
There are some occasions where it is not usually appropriate to use psychometric
assessment. For instance, it is inappropriate that such tools be used
for making redundancy decisions, since direct information on job performance
should already be available (such tools can be valuable in making re-deployment
decisions or in outplacement counselling however). Similarly, an organisation
may want to restrict the use of some assessments to counselling or development
applications.
Assessment tools are best used in decision making in conjunction with
other relevant information. In a promotion decision, the results from
psychometric assessments may be integrated with interview performance,
the candidate’s track record and a manager’s recommendations
to provide the best information about an individual’s suitability.
Use of a single assessment result alone should be avoided whenever possible.
Who Should Use Psychometric Instruments?
Knowledge and experience are required to use psychometric tools effectively. It is recognised throughout the world that the use of psychometric instruments by unsuitably qualified individuals can have very detrimental effects. To access materials requires both formal training in their use and often specialist training for the specific instruments in question. Qualified users should ensure that materials are only used appropriately and are not used by untrained people or for a purpose for which they were not intended. It is also their responsibility to work within the confines of their own expertise and to recognise when refresher training, skills updates or expert advice is needed.
Using Psychometric Instruments
Choice of Instrument
It is important that whenever instruments are chosen there is written
documentation of the reasoning behind the choice. This may include copies
of job analysis reports, job descriptions, person specifications, validation
studies, etc. If the relevance of the particular measure is challenged,
such evidence supports the instrument choice, shows the care taken and
helps ensure users do not take inappropriate short cuts. The importance
of job analysis in determining the knowledge, skills and abilities required
for effective on the job performance is magnified by the Human Rights
Act 1993 which makes it unlawful to make a selection decision on the basis
of information that may be discriminatory or is non-job relevant.
There are three main aspects to consider when choosing a psychometric
instrument: the content of the instrument; the level the instrument is
aimed at; and its psycho-metric qualities (Cronbach, 1984).
i) Instrument Content:
Whenever psychometric assessments are used it is vital that there is a
match between the skills and characteristics measured and the job and
organisational demands. This is reinforced in the Pre-employment Guidelines
Based on the Human Rights Act 1993, Section 3, where it is stated that
“Employers should only request information that is clearly relevant
to the requirements of the employment and to the applicant’s ability
to do the job”. For instance, the instrument should not require
understanding of complex vocabulary or performance at speed unless these
are relevant to the job. This is particularly important when selection
or promotion decisions are based on assessment results.
Objective Job Analysis is the best way to determine the skills and attributes
required for a particular job. These skills are then matched to appropriate
assessment tools. The more detailed the analysis of the job and the closer
the match between the attribute required on the job and that measured
by the instrument, the higher the content validity of the tool. Wherever
possible this approach should be supported by empirical research relating
assessment results to actual job performance. This information will allow
users to refine their use of psychometric tools and answer the question
- how relevant are they? In large scale assessment procedures it may be
appropriate to perform criterion related validation studies before utilising
a psychometric instrument.
The second aspect of instrument content that we should be aware of when
matching assessment materials to jobs is the context in which the skill
is measured. This should, as far as possible, reflect the type of content
found in the job. For example, a typing test should require the typing
of material similar to that required on the job. However, care must be
taken not to include material requiring knowledge specific to the organisation
that would, for instance, put external applicants at an unfair disadvantage.
Content of an instrument that is of a more general nature should be equally
accessible to all applicant groups, men and women and ethnic minorities.
For instance, in the typing test, if a job relevant text is too technical
for an external applicant to deal with before training, more general tests
should be used.
ii) Instrument Level:
The level of difficulty at which the skill or attribute is measured should
be appropriate to the job. An instrument which is too easy or difficult
will not differentiate between individuals with good and poor potential.
In selection the level of the instrument used should also be appropriate
for the likely applicant pool. If the general level of applicants is below
the level of standard required for the job, employers should consider
what they can do to attract better applicants. Training or job redesign
options also need to be considered. If there is a tendency for individuals
from one particular ethnic group or gender to fail to meet the required
standard, Section 73 of the Human Rights Act 1993 sometimes allows positive
action targeted at this group in order that they may “achieve an
equal place with other members of the community”. This may take
the form of special training programmes.
iii) Psychometric Qualities:
Assessment instruments should be psychometrically sound. The relevant information and statistics for judging the instruments should appear in the user manual. These should include:
- Specification of the skill or attribute the instrument measures -
we need to know that the instrument we are using is actually assessing
the type and level of skill or attribute required on the job.
- Description of groups for which the instrument is appropriate (educational
background, work experience etc) - we need to ensure that the instruments
we use are appropriate for the individuals being assessed. This can
be done by referring to the biographical details of the norm group chosen
to make comparisons. For example, we would not give applicants to a
clerical position a verbal reasoning test designed for university graduates.
- Details of the development process - it is important to find out whether
the instruments we use have been developed in a rigorous manner. All
instruments should be developed on the basis of a thorough job analysis
to ensure that they are assessing skills, knowledge or abilities necessary
for effective on the job performance in the occupational area for which
they are designed. For instance, each question within an aptitude test
needs to have been carefully reviewed and trailed to ensure that it
is interpreted in the same manner by all candidates and is non-biased.
To this end, the developers of psychometric tools should have reviewed
every single item within each instrument to eliminate those that do
not appear to be rationally related to other items, those that are ambiguous
in any manner or those that are judged unsuitable in any other way.
The developers should also conduct a number of trials for each tool
to ensure that they are both reliable and valid
- Instrument reliability statistics - when interpreting assessment
results, it is very important to know how consistent and therefore,
how accurate the scores are. Psychometric instruments are measuring
devices in their own right. They have their own errors of measurement
which may cause scores to vary from one session to another. The assessee’s
effort and attention may vary from time to time and changes in experience
or health may have an effect. The way the instrument is administered
can also influence the individual(s) being assessed. The content of
the instrument itself can also be a contributor to error, although this
can be reduced by eliminating poor items and ambiguous questions or
statements. It is important to know how much inconsistency there is
in any instrument so that this can be allowed for in the interpretation
of the results.
- Evidence of instrument validity - the validity of an instrument is
concerned with the extent to which it actually measures what it has
been designed to measure. For example, when thinking about the validity
of a personality questionnaire for measuring say, customer service style,
we would be addressing the question “how well does this questionnaire
actually measure the style required on the job?” We also need
to know that the instruments we use for an occupational group are assessing
skills related to successful on the job performance for that group.
The stronger the relationship between instrument performance and on
the job performance in a particular skill or aptitude area, the greater
the utility of the instrument. The validity of an instrument should
be the primary consideration of any user when choosing an assessment
tool.
- Steps to guard against ethnic and gender bias - thorough analysis
of an instrument’s content should take place in order to ensure
that questions are free from sexist language and are equally meaningful
to all ethnic groups to whom they may be administered. Research shows
that for some occupational assessments differences are observed in the
performance of gender and ethnic groups. The reasons for these differences
are extremely complex; differences in applicants socio-economic backgrounds,
their educational experience and training opportunities are likely to
be reflected in their assessment results. To ensure that an instrument
is not unfairly discriminating between ethnic or gender groups an assessment
of on-the-job differences in performance should be conducted. If there
are differences in assessment, but not on-the-job performance for each
of the groups, then the use of the instrument could be unlawful (Hunter,
Schmidt, and Hunter, 1979).
- Representative and relevant norm groups
Do not judge an instrument solely by how widely it is used. An instrument can become out-dated and companies sometimes use inappropriate and out-dated assessment tools.
Preparation of Candidates and Administration of Assessment Tools
Standardisation is the fundamental concept in the utility of tests (Mischel,
1986). It is the standardised conditions, instructions, time, content,
scoring and interpretation which make psychometric tools (or any assessment
tool or methodology) objective.
i) Use Of Practice Materials:
Some candidates may be unfamiliar with assessment processes so it may
be difficult for them to perform at their best. Others may find the assessment
situation very stressful. Ethnic minority candidates in particular, may
perhaps, under-perform because of the effects of educational disadvantage
or race discrimination. Older candidates and those with less educational
experience are also likely to suffer these sorts of problems. Practice
items at the beginning of an assessment procedure can reduce the bias
that may arise from differential assessment sophistication, helping some
people but not others. They can also reduce nervousness by allowing a
candidate to gain confidence in his/her ability to perform well in the
assessment. If possible, candidates should be notified a week in advance
that they will be assessed. Examples or descriptions of what assessment
instruments will be like (practice leaflets) should be provided, so that
candidates can familiarise themselves with the type of tasks involved
(Kellett, Fletcher, Callen, & Geary1994). Such practice increases
the effectiveness of the assessment proper by giving an accurate measure
of a candidate’s style or ability.
ii) Administration of Psychometric Instruments:
The administration instructions are extremely important and must always
be strictly adhered to. Only qualified persons should administer assessment
tools. Abuse of procedures described in the instrument manual can lead
to bias and possible unlawful discrimination. Special care should be taken
with people whose first language is not English to ensure that they have
understood the administration instructions properly. Some assessments
which are fair for native English speakers will present problems for people
with a lesser command of the English language. Instruments requiring reading
skills when these are not an integral part of the job are particularly
likely to be unfair. Where possible, such candidates should be assessed
in their native language. There are a number of aspects to instrument
administration that should be standardised, this is illustrated below:
Standardisation:
An encouraging attitude on the part of the administrator is always desirable, but it is particularly important to establish rapport with individuals who might lack confidence or who feel anxious about the assessment. The introduction to the assessment session is an important part of the administration procedure and instructions should be clear and not rushed. It allows the establishment of this rapport and should be conducted in a serious yet friendly manner. Information should be provided during instrument administration on the following:
- Why instruments are being used and how they fit into the assessment
procedure.
- The intended recipients of the information.
- The name and address of the organisation responsible for conducting
the assessments.
- The name and address of the organisation who will hold the assessment
results.
- That the completion of the assessment is not compulsory.
- The implications, if any, should the candidate choose not to complete
the assessment.
- That they will receive meaningful feedback relating to their performance
on all assessments conducted.
There should be an opportunity for candidates to ask general questions before the formal assessment procedure starts.
Confidentiality and Storage
The Role of Feedback
The Privacy Act 1993 requires that whenever assessment results are used
assessors should be honest and open with candidates about why the instruments
are being used and what will happen to the results. Members of the Psychological
Society are also bound by the Code of Ethics to obtain the informed consent
of the individuals to be assessed when undertaking a psychological assessment.
As such, individuals must be informed of the right to know the content
of psychological assessment reports and in reporting findings, psychologists
must endeavour to ensure that appropriate explanations of the findings
and their interpretations are given. All individuals assessed should thus,
be offered meaningful feedback of their results as soon after the assessment
process as possible. Personality and motivation questionnaire feedback
is critical and will often enhance the interpreters own understanding
of assessment results.
Feedback does not need to be lengthy, indeed with a large number of applicants
this might be very time consuming. A face to face interview is preferred,
but telephone feedback may be the only option in some circumstances. Feedback
should be given by qualified users and should be an open and accurate
two-way process. The NZPS guidelines do not sanction the release of uninterpreted
data from assessments to persons untrained in their use and interpretation,
profile charts may be shown to respondents but they should not be given
copies to take away. A short narrative summary may be provided if desired.
This is particularly useful where assessment is for counselling and development
purposes.
Computer generated or narrative reports can support, but should not replace,
the feedback interview. Some may be suitable to give to respondents, but
many are intended as aids to interpretation to the trained instrument
user and could easily be misinterpreted by others. Users should follow
guidelines provided by the author or publisher of such systems.
Under Information Privacy Principle 8 of the Privacy Act 1993, personal
information gathered and held about an individual should not be used without
ensuring that the information is accurate, up to date, complete, relevant
and not misleading. Feedback interviews are an important part of this
validation process.
Assessment Results
Assessment results, like all personal information, should be stored
with the strictest regard to confidentiality. Access should be restricted
to those with a need to know and in accordance with what has been agreed
with the respondent during administration and feedback. Persons who are
untrained should not be allowed access to raw data from assessments, but
only to clear interpretations of those results.
Individuals do change and develop and so psychometric data can become
less accurate over time. Instrument scores should therefore not be kept
on file indefinitely. The time period for which scores are valid will
differ depending on the nature, the measures and the particular use made
of them. Care should be taken with the use of results over 6 to 12 months
old for selection purposes. Little reliance should be put on results over
2 years old for any purpose. The Privacy Act 1993, requires that all personal
information, including assessment results, should be protected by such
security safeguards as it is reasonable to take to ensure against (1)
loss and (2) unauthorised access, use, modification or disclosure.
Materials
The security of materials is paramount. Free circulation leads to over familiarity and devalues psychometric instruments. Responsible instrument publishers only supply materials to trained users who, in turn, must ensure untrained users do not gain access to them. Within an organisation decisions should be taken about who should hold assessment materials and who should have access. It may not be desirable for all users to have access to all materials. Central storage can help prevent unnecessary duplication of materials but may not be practical in decentralised organisations. An organisation must supply instrument users with appropriate storage space where the materials can be kept under lock and key. It is highly desirable that all materials are logged in and out of storage when used. This helps ensure materials are not carelessly left lying around or misplaced. Failure to keep track of materials can be expensive where replacements have to be purchased or annual lease fees paid on missing booklets.
Summary and Conclusion
Research and legislative precedent is firmly entrenching the importance
of rigorous job analysis to justify the use and choice of tests. Our ethical
and professional obligations as psychologists are clearly laid out and
oblige us to only use tools for which we are trained. Validity and reliability
and appropriateness of the items should underlie our evaluation of tests.
Standardisation is the key principle behind the application of psychometric
tools. As practitioners this does not always come easily to us, but being
rigorous must be foremost in our minds.
References
Cronbach, L.J. (1984). Essentials of Psychological Testing (4th ed.).
Harper & Row New York.
Hunter, J.E., Schmidt, F.L. & Hunter, K. (1979). Different validity
of employment tests by race: A comprehensive review and analysis. Psychological
Bulletin, 86, 721-735.
Kellett, D., Fletcher, S., Callen, A., & Geary, B. (1994). Fair Testing:
The Case of Bristish Rail. The Psychologist, January, 26 - 29.
Mischel, W. (1986). An Introduction to Personality. CBS College Publishing.
Japan.
Uniform Guidelines on Employee Selection Procedures. (1985). Equal Employment
Opportunities Commission, Department of Labour & The Office of Personnel
Management. 29CFR, Section 1607, Washington DC.