"Proper use of well-constructed and validated tests provides a better basis for making some important decisions about individuals..."
Standards So states the Committee on Ability Testing of the National Research Council as quoted from Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, National Council on Measurement in Education, 1985 edition, p.1). (Standards will be used to refer to that text henceforth.) That publication also points out that not all tests are well-developed, nor all testing practices wise and beneficial. Therefore, establishing the validity and reliability of the testing procedure -- that is, both the test instrument and test administration -- is an important first step before the test can be employed with confidence.
Validity & Reliability Without first-hand experience with the predictive ability of a test, external measures of validity and reliability help the decision maker to choose between alternative tests. The term validity is used to convey the idea that the test procedure measures what it purports to measure. For example, a ruler may be employed in a more valid measurement of length than a thermometer. Face validity, a measurement of validity based upon readily accepted judgments, is a property of many inventories which ask questions that are directly related to the trait being measured. For example, for the measurement of the trait ‘sociability,’ the question “Do you enjoy meeting new people,” would be generally accepted as face valid. This approach was taken in the development of the REMBRANDT PORTRAIT®.
Another factor that affects validity is respondent biases. Asking questions about traits directly (such as, “How social are you?”) can elicit consciously or unconsciously biased responses.
Instead, inventories such as the REMBRANDT PORTRAIT® are employed to estimate personality variables by examining a set of responses to specific statements consonant with the trait. For example, instead of asking “How social are you?”, the statements might include: “I make friends easily,” “I enjoy a lot of company,” and so forth. These statements are presented as alternatives to other positive-quality statements, such as “I do my work responsibly,” “I do not tire easily,” and the like. Therefore, in choosing one positive trait over another, it is very difficult for an individual to bias the results in favor of a particular quality without depreciating another positive quality. The REMBRANDT PORTRAIT® makes use of this method, selecting from among competing positive traits, to minimize respondent biases.
The other concern of the decision maker, reliability, also bears upon the utility of an inventory. Reliability of a test concerns the degree of test-measurement error. A common measurement of reliability is the consistency between two administrations of the inventory to the same people. Factors contributing to low reliability in tests include, but are not limited to the following: (1) differences in the way the test instrument was administered, (2) interactions between the test giver and the test taker, (3) cultural differences, (4) differences in the evaluations and interpretations of test scoring.
Choosing an inventory In deciding to use any particular test to aid in a decision process, the decision maker must either be familiar with the use of the test or be familiar with the standards to which the test was validated. Consistent with Standards, we note that it is not appropriate for test developers or users to state that a test satisfies or follows the standards set forth by that publication. Rather, based upon both the validation data available and the experience with the test, professional judgments must be made.
REMBRANDT PORTRAIT® The REMBRANDT PORTRAIT® is a computer-based test of fourteen personality traits which are used for evaluating candidates for employment. Because it is administered by a PC-computer rather than by different administrators, variability both within and between administrators, which might detrimentally affect reliability and validity, is eliminated. Therefore, by employing a computer to administer the test, it may be more reliable and, therefore, possibly more valid than it would be otherwise. This document provides an overall examination of the REMBRANDT PORTRAIT®, administered to persons already employed, and compares their REMBRANDT PORTRAIT® -scores with self evaluations given by means of a standardly administered inventory. By comparing REMBRANDT PORTRAIT® scores with face-valid inventory scores in persons who have no reason to conceal their true personality traits, the REMBRANDT PORTRAIT® creators tried to obtain data that would demonstrate validity by reference to the inventory scores. To compare the REMBRANDT PORTRAIT® results to standardly administered inventory results, correlations were computed for each of the ten primary items of the REMBRANDT PORTRAIT®. In addition, to demonstrate the extent to which the ten REMBRANDT PORTRAIT® traits reflected separate, distinct measures of personality, a factor analysis was employed.
The ten primary traits of the REMBRANDT PORTRAIT© correspond to the following characteristics:
TRAIT -- CHARACTERISTICS
- Aggressiveness -- willing to criticize, confrontative
- Assertiveness -- leader, controls situation, less self-doubts
- Caution (approach to decisions) -- minimizes risks, thoughtful decision maker
- Correctness (sensitivity to rules / guidelines) -- follows rules, does things the proper way
- Energy (stamina) -- works hard, lots of energy
- Friendliness (sociability) -- enjoys making friends, likes to mix socially
- Nurturance (service motivation) -- helpful, understanding
- Pace / Urgency -- works faster, intense, impatient
- Responsibility -- reliable, persistent, thorough
- Structure (self-discipline) -- orderly, plans things, sticks to plan
NeuroCommunication Research Laboratories, Inc. (NCRL) analyzed personality-trait components of the REMBRANDT PORTRAIT® in order to obtain information about the REMBRANDT PORTRAIT® as an employment assessment tool. NCRL was provided with data collected during 1989 / 1990 from 143 employed individuals who completed both the REMBRANDT PORTRAIT® inventory and also a traditionally administered, “paper-and-pencil” personality inventory (referred to as Inventory henceforth). Paired data were provided on each of the 10 primary personality traits. The study population was derived from a variety of industries, job-levels, and educational levels. There was a mix of age-levels, sexes, and racial groups.
The REMBRANDT PORTRAIT® data were taken directly from the REMBRANDT PORTRAIT® computer program, which assigned numerical values to statements selected by respondents interacting with the REMBRANDT PORTRAIT® computer program. To determine each personality trait, multiple statements were utilized. The REMBRANDT PORTRAIT® computer program categorized responses according the personality trait that they reflected. The sum of the numerical values for each personality trait was the raw score for that trait.
The Inventory presented statements describing hypothetical individuals who would be considered to have a high degree or a low degree of a personality trait. Respondents were asked to highlight the area of an unnumbered, 50-division scale which corresponded to their self-perception on the particular trait. Data were converted by linear transformation into a 0100 scale. The center of the highlighted area was used in the analysis to represent self perception.
Pearson Product-Moment Correlations were computed between the paired personality traits from the REMBRANDT PORTRAIT® and Inventory. The correlation coefficients for each trait follow.
Each of the correlation coefficients is statistically significantly positive at better than the 99% confidence level (p<.001, two-tail).
The following conclusion can be drawn from these analyses:
The computer-based administration method of the REMBRANDT PORTRAIT® provides a personality inventory that was generally comparable to a traditionally administered “paper-and-pencil” inventory in a sample of employed persons.
FactorsIn order to determine whether the REMBRANDT PORTRAIT® traits measure distinct constructs, NCRL performed a factor analysis on the REMBRANDT PORTRAIT® personality traits. The procedure employed the method of Principal Components with Varimax rotation. To allow full freedom of effect, no restrictions were placed on the components before rotation. The results of this analysis yielded 9 factors (i.e., components) which accounted for approximately 99% of the variance. Eight of the factors corresponded specifically to a personality trait, that is, the factors loaded highly on one trait (.88.99) and low on all of the other traits (<.49). The percentage of total variance explained by each of the trait-related factors is presented below:
The other factor loaded highly on two of the remaining traits.
Correctness / Aggressiveness -- 14.87
A similar factor analysis was computed on the Inventory. The varimax rotation yield 10 factors yielding approximately 10% explained variance each (range: 9.9-10.1). Each factor loaded highly (.93 or above) on only one personality trait and low (.18 or below) on all other traits.
The following general conclusion can be drawn from these factor analyses:
The 10 primary traits included in the REMBRANDT PORTRAIT® seem to be related to 8 distinct personality constructs and one other construct consisting of 2 traits
Strength of Single Factored ConstructsThis result (8 single-factored constructs) shows a strength Factored Constructs of the REMBRANDT PORTRAIT®, yet this result contrasts with other results found for traditionally employed tests, many of which are multifactorial. Constructs which are multifactorial are undesirable, because of two pitfalls. The two pitfalls are: (1) Internal consistency reliability of the overall construct is weakened; (2) The degree of validity can be compromised.
Unlike some other employment inventories, most of the traits on the REMBRANDT PORTRAIT® are single factored. The single multifactorial construct of the REMBRANDT PORTRAIT® is two factored (combining correctness and aggressiveness), which would allow deviations in the anticipated association (i.e., more aggressive is less correct on details) to be separated by subsequent interviewing.
REMBRANDT PORTRAIT® Consistency With Other InventoriesThere does seem to be a strong correspondence of the traits measured in the REMBRANDT PORTRAIT® and traits measured in other established personality inventories. For illustrate purposes, REMBRANDT PORTRAIT® traits are listed which generally correspond with two better known personality inventories, The Personality Research Form (PRF) and the Edwards Personality Inventory (EPI).
These inventories were developed to provide an instrument for measuring broadly relevant personality traits for use in education, business, and counseling, a goal consistent with that of the REMBRANDT PORTRAIT®.
The PRF has 20 traits of which 9 correspond with the 10 primary traits of the REMBRANDT PORTRAIT®. The corresponding traits of the PRF are further described in the following table. These descriptive adjectives are the ones presented in the PRF manual.
The EPI has over 50 scales, several of which relate to a common personality construct. All of the 10 primary traits of the REMBRANDT PORTRAIT® relate to one or several of the EPI scales.
Table 1. REMBRANDT PORTRAIT® and PRF Adjectives
|Assertiveness||Autonomy (self reliant, individualistic)|
|Energy||Achievement (industrious, ambitious)|
|Friendliness||Affiliation (warm, courteous)|
|Caution||Harm avoidance (fearful, cautious)|
|Correctness||Social Recognition (approval seeking)|
Table 2. REMBRANDT PORTRAIT® and EPI Adjectives
|Assertiveness||Is a Leader, Self Confident, Independent|
|Aggressiveness||Persistent, Critical of Others, Becomes Angry, Motivated to Succeed|
|Energy||Is a Hard Worker, Active|
|Pace/Urgency||Plans Work Efficiently|
|Friendliness||Makes Friends Easily|
|Caution||Worries About Making A Good Impression, Anxious About Performance|
|Correctness||Likes a Set Routine, Conforms|
|Structure||Plans and Organizes Things, Logical|
Other personality inventories, such as the Gordon Personal Profile and Cattell’s Sixteen Personality Factors Questionnaire (16 PF) also have traits which generally correspond with the REMBRANDT PORTRAIT®.
In summary, the REMBRANDT PORTRAIT® is a rapid personality assessment tool, which yields ten primary measurement scales. The REMBRANDT PORTRAIT® is objectively administered and scored by computer. Potential employers can personalize the test for their particular situation, instead of applying one set of norms for all. This analysis indicates that there is very little overlap in what the ten primary scales measure. Additionally, the REMBRANDT PORTRAIT® traits correspond with a traditionally administered “paper-and-pencil” inventory. The traits measured by the REMBRANDT PORTRAIT® are likely to be useful for screening for all employment situations.
NeuroCommunication Research Laboratories (NCRL), an independent laboratory, functioned as a consultant to the creators of the REMBRANDT PORTRAIT® to prepare this document. Though not directly involved with the creation of the REMBRANDT PORTRAIT®, the study design, the data collection, the preliminary data reduction, or the data entry, NCRL performed the statistical analyses and the interpretations that are presented in this document.
VALIDATION STUDY RESULTS CHART 12/29/90
Degree of match between self assessment questionnaire and REMBRANDT PORTRAIT® results: 143 Individuals
|[individuals claiming more]||[Individuals claiming less]|
CONCLUSION: About two-thirds of the traits measured are within 5%
About 85% of the traits measured are within 15%
Test / Re-test Reliability
Another phase of this study was to determine the tests’ test – re-test reliability. It is important for any employment measuring tool to consistently measure an individual over time. The 143 people involved in this study were administered the REMBRANDT PORTRAIT® several times over a period of 18 months.
Test / Re-test reliability coefficients:
|3 months||6 months||12 months||18 months|
The REMBRANDT PORTRAIT® has strong internal stability resulting in strong correlation between the original test and subsequent tests.