Criteria Selection and Methodology in Comparative Scaling Approaches Study

The selection of the criteria for this study was primarily guided by considerations that were practical: availability in the data set in existence. In addition, we had the intentions of working with a criterion that was familiar and domains of prediction such that the study`s primary focus would be on the comparison of approaches for scaling and not on development and testing of theories. College Grade Point Average (GPA) was the main criterion for the study and this was collected from records that were obtained from the school which were not very old. Also included were two concurrent criteria. While a high-school leadership scale was derived from (Fackler and Malmberg, 2016), school records were used to get the high school GPA. The authors who developed the leadership scale developed rational scales and further put their relations with adolescent leadership into scrutiny. The scale included four BQ40 items that were reflective of the activities of leadership for instance, direction of others in group activities, school politics participation, holding of positions of leadership, having influence on others, and other students describing you as a leader. .42 was the internal coefficient of consistency.

For the present study, two rational scale sets were developed one for leadership in high school and another for college and high school GPA. We had expectations that there were several variables that would have a positive relation to the GPA like self-esteem, motivation, verbal skills and mathematics. As such, for the criterion of the GPA, the following construct scales were generated: motivation for achievement, scientific interests/ quantitative skills, work ethic, adaptation to the institution (adjustment to college) and self-esteem. It has been observed that often, there exists a correlation between college grades and high school grades. That prompted the addition of a high-school achievement scale, that is, course specific grades into the prediction of college GPA. This findings are suggestive of the importance of social variables and motivation for leadership in high school. These variables consistent relationship with variables to leadership indexes has been shown by the broader literature of leadership. In line with that, the rational scales that tap into motivation for achievement, verbal skills, self-esteem, work ethic, dominance/persuasion, adjustment to social surroundings and also independence were generated for the criterion of leadership.

Rational scales were developed through the use of modified versions of the procedures that were described in (Hamilton, Shih and Mohammed, 2016). Panels of three organisational psychologists that had experience in dealing with background data of up to five years were convened. Members of the panel did an examination of a conceptual definition and that aided in the development of the construct`s consensual understanding for every data construct. After that, the panel members were asked to carry out independent selections from the BQ40 that would be used for marking the different constructs. In carrying out the item selection panel, members of the panel were requested to thoroughly think about reactions, behaviour and outcomes in different situations that would be indications of manifestations that were direct of the different constructs at hand or which would contribute to the construct at hands development. In the study by (de Zoete et al. 2017), there were some of the construct`s scales. Everyone was provided with a list of items that were in the scale. For all the different scales, the process was repeated. The rational scales that were developed for the criteria of the GPA had reliabilities that ranged from .48 to .76 and all these scales had a total of 52 items. In table 1 below, there is a list of the rational scales, providing the number of items that every scale has, items in the samples and Cronbach alphas.

Empirical criterion keying involves the procedural development of the key scoring in terms of an external criterion whereby, we selected the items that were supposed to be retained together with the assigning of scoring weights to different responses (Cucina et al. 2019). The most commonly used method in the scoring of background data in selection of personnel is empirical keying. Generation of empirical keys happens through combination and weighing of items with their relation to the interest criterion being the main basis of their relationship. Empirical keys capitalise on item-criterion relations for maximisation of the size of the coefficients of validity. That goes a long way in making the validity that is related to the criterion of the empirical keys to be quite strong. Subtle and non-obvious relationships could also be identified by empirical keys between items and the criterion that may be missed by other different scoring techniques.

However, even with empirical keys having the capabilities of predicting, critics argue that initial validities` size is often inflated as a result of the fact that the keys capitalise on the chance relations that exist between items and the criterion followed in key development. Typically, validities decrease whenever cross-validation of keys happens or they are applied in different samples aiming at ensuring generality and stability. The use of samples for development that are not adequate in reflecting those samples to which the keys are applied is responsible for the reduction in addition poor person-to-predictor ratios.

There are different methods used by researchers for construction of empirical keys, like vertical percent, regression/correlation, men criterion, horizontal percent and also rare response. The available options for development of empirical keys are added by variations in each of the approaches. Different weighting and selection procedures are used in different keying methods; even with that, variance maximising procedures are used in most of the methods for identification of options and items that are the best in discriminating high performers from low performers.

There are numerous methods of key construction available and for this study, two were selected; a vertical-percent approach and a correlation approach. Research done in the past recommends that whenever the items to be used and measures of criterion are continuous, the development of the key should be guided by the existing correlation between each item. Pattern-of-response keying was selected as the correlational method for keying given that here, criteria and continuous items were applied. Establishment of criterion groups was not required in the keying procedure.

Participants

Owens and Schoenfeldt (1979) carried out a longitudinal study that was used as the basis of this study. To be specific, the study`s participants were made up of 540 freshmen from college (300 women and 240 men) from the immediate surroundings. On a random basis, the data was divided into samples of validation that were made up of about three quarters of the participants (301 women and 104 men). It is worth noting that while some of this data was used in studies that were conducted in the past, comparisons of the procedures of sampling that were used in this study were not addressed in the past. A 40-item background-data questionnaire was issued to the participants when the freshmen orientation was happening. The questionnaire was used in the development of the different rational keys and rational scales that were used in the study. The questionnaire was made up of different questions that were about general experiences on life in different areas in their day to day activities like family, school, hobbies and even friends.

Results

Before the onset of the comparison of different scoring methods` cross-validities and validities, some evidence is provided with regard to the meaningfulness of the criterion measures and the rational scales. The standard deviations, means and correlations of the high school GPA, college GPA, rational scales, and high-school scale of leadership are included in table 2. Large positive correlations were observed from high school achievements (specific classes` grades) GPA (r = .63) and college GPA (r = .42), work ethic showed correlations that were moderate (r=.36) and persuasive dominance and social adjustments showed small correlations (r=.05 and .03 respectively). There was moderate to large correlation between institutional adaptation and self-esteem (r=.33), motivation for achievement (r=.29) and some relatively small correlation with persuasive dominance and social adjustment (.09 and .03 respectively). Leadership in high school had the highest correlation with adjustment socially (.58), work ethic (.51), persuasive dominance (r=.46), self-esteem (r=.40) and motivation for achievement (r=.48) and showed rather small correlations with independence (r=.20). We were surprised that there were not large correlations between college and high school GPA with the scale for quantitative and scientific interests. The correlation would have perhaps been larger had we tapped into quantitative and scientific skills instead of the interests. However, in general, the patterns of discriminant and convergent relations among other different criteria and rational scales provide the initial evidence that what is being measured by the scales is what was intended to be measured.

Rational Scales Means, Standard Deviations and Correlations

References

Cucina, J.M., Vasilopoulos, N.L., Su, C., Busciglio, H.H., Cozma, I., DeCostanza, A.H., Martin, N.R. and Shaw, M.N., 2019. The Effects of Empirical Keying of Personality Measures on Faking and Criterion-Related Validity. Journal of Business and Psychology, 34(3), pp.337-356.

de Zoete, A., de Boer, M.R., van Tulder, M.W., Rubinstein, S.M., Underwood, M., Hayden, J.A., Kalter, J. and Ostelo, R., 2017. Rational and design of an individual participant data meta-analysis of spinal manipulative therapy for chronic low back pain—a protocol. Systematic reviews, 6(1), p.21.

Fackler, S. and Malmberg, L.E., 2016. Teachers' self-efficacy in 14 OECD countries: Teacher, student group, school and leadership effects. Teaching and Teacher Education, 56, pp.185-195.

Hamilton, K., Shih, S.I. and Mohammed, S., 2016. The development and validation of the rational and intuitive decision styles scale. Journal of personality assessment, 98(5), pp.523-535.

Owens, W.A. and Schoenfeldt, L.F., 1979. Toward a classification of persons. Journal of Applied Psychology, 64(5), p.569.

Discover additional insights on Content On Social Media by navigating to our other resources hub.