Validating the 360 Instrument

by Thuy Sindell, PhD. and Milo Sindell, MS.
Published on January 1, 2014

Background

In 2001, led by Gustavo Rabin, Ph.D., Founder and President of Skyline Group, we assembled a core research team from Stanford University, along with fellow academia colleagues from University of Massachusetts and University of Florida. The objective was to capture the core of the research performed through all other existing 360 assessments and distill the essence defined as the fundamental competencies that support or derail a career.

Instrument Development

The research identified 28 competencies across four core domains: Leading Self, Leading Others, Leading the Organization, and Managing Implementation/Execution as critical for leadership success and effectiveness in organizational settings.

There is a theoretical and empirical basis for the four domains and 28 competencies in terms of content validity. Since the technology of competency- based 360 feedback surveys is an outgrowth of the tradition of psychological testing, it’s natural to confuse the two kinds of assessments. Even though they’re both often referred to as “assessments,” they’re different kinds of tools, and the procedures for validating them are different.

Validity, Reliability and the Behavioral Descriptor Approach

Validity means one thing when evaluating a personality assessment or a psychological test, but it means something different when evaluating a tool that gives direct feedback about behavior — which is a different kind of assessment. Validity and reliability are important in terms of determining the usefulness of tests that measure intelligence, values, or psychological traits or characteristics. These are important aspects of people that cannot be observed or measured directly, so questions are asked to draw inferences about these aspects. Therefore, it’s important to determine whether the assessment really measure what it says it measures and to measure its reliability (e.g.., test-retest reliability or measuring internal consistency).

The Skyline 360 employs a different approach. Our approach and process is to gather and provide practical feedback on a specific set of observable behaviors. Unlike psychological instruments, the Skyline 360 items (i.e., behavioral descriptors for each of the 28 competency) themselves are the primary focus of the feedback. A number of the competency-based items include multiple statements and this is by design. The qualitative component and 1:1 discussion between the manager/coach and participant elucidates the specific areas of focus.

In term of validity, the Skyline 360 and the 28 competencies are derived from extensive theoretical and empirical research, thereby suggesting both content validity and concurrent validity (i.e., competencies related to leadership effectiveness and job performance). Measures of reliability (e.g., test-retest reliability or measures of internal consistency such as Cronbach’s alpha), which are routinely performed on psychological tests and employee/organizational surveys, are not appropriate for competency-based 360-degree feedback tools since significant variability among the raters are likely and expected.

In short, the end goal of 360-degree feedback is ultimately about employee development and improvement – including establishing action plans and meaningful goals, coaching and performance feedback, and developing more effective business leaders. Some important questions to consider when looking for a 360 degree solution:

1. Does this 360 instrument measure what it is supposed to be measuring (e.g., are the core competencies in the instrument and behavioral statements/descriptors) related to real-world performance on-the-job, leadership quality and effectiveness, and/or a desired employee behavior)?

2. Does the 360 instrument sufficiently measure what your organization needs it to measure?

3. Are the questions/items on the 360 clear and can you provide a response without having to guess at what the competency and concomitant behavioral descriptor is getting at?

4. Are the results specific and clear enough that a manager/coach and participant can identify strengths and areas for improvement, have a meaningful coaching and performance feedback discussion, and come up with a SMART goals and a specific plan of action for each?

The Skyline 360 has a number of unique features:

• High rate of response: To encourage participation among all raters, the Skyline 360 could be completed in less than 10 minutes. Spending more time writing comments is optional.
• No redundant questions: In terms of the specific items/questions, the Skyline 360 uses “behavioral descriptors” which includes 1 to 3 simple statements to describe the competency holistically, yet in a clear and concise way so that we do not need to ask the same questions multiple ways.
• Comprehensive: The 360 Feedback report includes all essential competencies and skills for effective leadership.
• Rate multiple people at the same time: Managers, or others who are asked to provide feedback to multiple leaders, can do so all at once enabling a comparative view and a time saving approach.
• Powerful normative data: The normative data allows the leader to know how his/her scores compare with other successful leaders, understanding if s/he is “ahead or behind the curve”.

As of September 2013, almost 5,000 executives have received feedback through the Skyline 360 Feedback, assuring an incredibly robust database that yields accurate and reliable normative data.