Validating the 360 Instrument
Background
In 2001, led by Gustavo Rabin, Ph.D., Founder and President of Skyline Group,
we assembled a core research team from Stanford University, along with fellow
academia colleagues from University of Massachusetts and University of
Florida. The objective was to capture the core of the research performed
through all other existing 360 assessments and distill the essence defined as
the fundamental competencies that support or derail a career.
Instrument Development
The research identified 28 competencies across four core domains: Leading
Self, Leading Others, Leading the Organization, and Managing
Implementation/Execution as critical for leadership success and effectiveness
in organizational settings.
There is a theoretical and empirical basis for the four domains and 28
competencies in terms of content validity. Since the technology of competency-
based 360 feedback surveys is an outgrowth of the tradition of psychological
testing, it’s natural to confuse the two kinds of assessments. Even though
they’re both often referred to as “assessments,” they’re different kinds of
tools, and the procedures for validating them are different.
Validity, Reliability and the Behavioral Descriptor Approach
Validity means one thing when evaluating a personality assessment or a
psychological test, but it means something different when evaluating a tool
that gives direct feedback about behavior — which is a different kind of
assessment. Validity and reliability are important in terms of determining the
usefulness of tests that measure intelligence, values, or psychological traits
or characteristics. These are important aspects of people that cannot be
observed or measured directly, so questions are asked to draw inferences about
these aspects. Therefore, it’s important to determine whether the assessment
really measure what it says it measures and to measure its reliability (e.g..,
test-retest reliability or measuring internal consistency).
The Skyline 360 employs a different approach. Our approach and process is to
gather and provide practical feedback on a specific set of observable
behaviors. Unlike psychological instruments, the Skyline 360 items (i.e.,
behavioral descriptors for each of the 28 competency) themselves are the
primary focus of the feedback. A number of the competency-based items include
multiple statements and this is by design. The qualitative component and 1:1
discussion between the manager/coach and participant elucidates the specific
areas of focus.
In term of validity, the Skyline 360 and the 28 competencies are derived from
extensive theoretical and empirical research, thereby suggesting both content
validity and concurrent validity (i.e., competencies related to leadership
effectiveness and job performance). Measures of reliability (e.g., test-retest
reliability or measures of internal consistency such as Cronbach’s alpha),
which are routinely performed on psychological tests and
employee/organizational surveys, are not appropriate for competency-based
360-degree feedback tools since significant variability among the raters are
likely and expected.
In short, the end goal of 360-degree feedback is ultimately about employee
development and improvement – including establishing action plans and
meaningful goals, coaching and performance feedback, and developing more
effective business leaders. Some important questions to consider when looking
for a 360 degree solution:
1. Does this 360 instrument measure what it is supposed to be measuring
(e.g., are the core competencies in the instrument and behavioral
statements/descriptors) related to real-world performance on-the-job,
leadership quality and effectiveness, and/or a desired employee behavior)?
2. Does the 360 instrument sufficiently measure what your organization needs
it to measure?
3. Are the questions/items on the 360 clear and can you provide a response
without having to guess at what the competency and concomitant behavioral
descriptor is getting at?
4. Are the results specific and clear enough that a manager/coach and
participant can identify strengths and areas for improvement, have a
meaningful coaching and performance feedback discussion, and come up with a
SMART goals and a specific plan of action for each?
The Skyline 360 has a number of unique features:
- High rate of response: To encourage participation among all raters, the Skyline 360 could be completed in less than 10 minutes. Spending more time writing comments is optional.
- No redundant questions: In terms of the specific items/questions, the Skyline 360 uses “behavioral descriptors” which includes 1 to 3 simple statements to describe the competency holistically, yet in a clear and concise way so that we do not need to ask the same questions multiple ways.
- Comprehensive: The 360 Feedback report includes all essential competencies and skills for effective leadership.
- Rate multiple people at the same time: Managers, or others who are asked to provide feedback to multiple leaders, can do so all at once enabling a comparative view and a time saving approach.
- Powerful normative data: The normative data allows the leader to know how his/her scores compare with other successful leaders, understanding if s/he is “ahead or behind the curve”.
As of September 2013, almost 5,000 executives have received feedback through
the Skyline 360 Feedback, assuring an incredibly robust database that yields
accurate and reliable normative data.