Generality Maximization

Information which is undependable is not a fact. To say something is a fact is to say that it can be made to reoccur or be replicated, i.e., it must be reliable to be a fact. We know that if we follow a key peck with food then the response rate will increase. This effect occurs each time we do it. To say something is applicable is to say the information applies to a new situation of interest or that it generalizes to a new situation. This is referred to as external validity. Not only will following a key peck with food increase its rate, but following fist fights with cheers will increase their rate. In addition to reliability then, it is necessary that functional relationships or knowledge have generality. Information which is not repeatable or does not generalize to other situations is worthless. The test of reliability is the ability to be directly replicated or repeated under exactly the same conditions. The test of generality is the ability of a relationship to be systematically replicated or repeated in somewhat different situations. As everything in science, generality, in the last analysis, is empirically determined. If the functional relationship predicts in a new situation, then it is said to generalize.

The ability to generalize information from one situation to another is a function of several factors. The reliability of the original information; the paradigm's validity; your understanding of the paradigm and the true determinants of the behavior and the relevant details of the situations in question; and the similarity between the original source of the data and the situation to which it is to be applied.

There are both similarities and differences between the terms "stimulus generalization" and "generality of a functional relationship." Stimulus generalization is the description of the fact that an organism behaves in a similar way to similar stimuli and that the more different the stimuli the more different the behavior. The generality of a finding refers to the degree to which a functional relationship obtained in one situation is able to predict the obtained relationship in a new situation.

Keep in mind that we are not really interested in the "generality" of individual events but rather in the generality of functional relationships. We are not interested in the fact that responding occurs to WUZ about the same as it did to TUZ but rather that distributed practice helps in learning nonsense syllables and in learning other tasks.

All individuals together are called the population whereas a subset of the population is considered a sample. Typically a sample is measured and the information such as "response allocation matches reinforcement allocation" is taken as an indicator of that relationship in the population. If information is applied to a new situation we speak of that situation as a target, and the situation which was used to generate information applicable to that target as a model.

All the issues discussed in the section on reliability maximization are obviously appropriate in assuring generality. If two observers cannot agree on the nature of a functional relationship in the original situation, it is unlikely that agreement will occur from the original situation to a target situation. Therefore we can increase our expectation of generality by maximizing reliability.

If a functional relationship is obtained and is then applied to itself, it, of course, applies; it is simple description. If the original situation changes very slightly between the time the relationship is identified and then applied to itself the information will probably still correctly describe or predict. However, it can be seen that the less similar the original and target situation the less likely that the finding will apply to the dissimilar target. We may be able to increase the likelihood that information gained in one situation is applicable to another situation by making the two situations as similar as possible, hoping that the elements which are different are not important, and hoping that the factors which are the same are important. This is better than nothing but not entirely satisfactory. First, it may not work. For example, humans are similar to humans and fish are different from humans, however voting patterns of humans earning less than $5000 per year do not predict voting patterns of humans earning over $50,000 per year; whereas the effects on a fish of being dropped onto the surface of the sun predicts very well what will happen to humans. Secondly, it represents a rather passive, timid and superstitious approach to science. It deliberately minimizes the opportunity to learn why things work the way they do by minimizing the difference between the model and target. In the extreme, we would maximize similarity by using each patient as both the model and the target for himself. That is nothing less than the abandonment of science.

It is obvious that the issue revolves around the definition of the word "similar." If the model and target are similar with respect to the controlling variables of interest then the functional relationship will generalize. If they are similar with respect to irrelevant dimensions and dissimilar with respect to relevant dimensions, then the findings will not generalize. If we consider two groups similar because the effect of interest will generalize, then we cannot turn around and claim that if the groups are similar then the effect will generalize. This is a tautology. Basing expectation of generality on naively perceived similarity represents either an unsubstantiated hope or a tautology, neither of which provide any meaningful help. Simply because the research was done on primates we are not assured that the functional relationship will generalize to humans. Simply because the functional relationships were obtained from worms we are not assured that those findings will not generalize to humans.

Paradigmatic validity refers to whether or not different situations are actually related to each other the way your paradigm suggests that they are. Whether or not a functional relationship obtained in one situation will generalize to some target situation depends on whether or not the true causes of the effects in the original situation are going to reoccur in the necessary way in the target. Your ability to predict whether or not the effect will generalize depends on your understanding of the actual causes of the behavior in the original situation and your correct understanding of whether or not those same factors are operative in the target situation. You ability to generalize the finding is therefore affected by the validity of your paradigm. It is important to realize that two elements are needed to predict whether or not an effect will generalize to a new situation: 1) exactly why it occurred in the original situation and, 2) how the new situation relates to the original situation with respect to the relevant details. In an entirely alien reality you would have great difficulty predicting what would happen in new situations even if you knew why things happened in the original situation. This is because you would have a very poor idea of the circumstances in which to expect those same controlling factors again. Rather like Alice in Wonderland. You cannot make accurate predictions unless you know the rules by which the situations are connected. A famous example of true rules being other than what they appear on the surface would be the Abbott and Costello comedy routine "Who's on First."

In sum, knowledge can be generalized on the basis of simple naive notions of similarity, or on the basis of understanding what's going on. This has led to two research strategies: 1) research emphasizing similarity to maximize generality, and 2) research emphasizing the understanding of the mechanisms of action and the control of confounding factors to maximize generality. "Similarity" research is often done in the "real" world with "real" people or at least primates while "understanding" research is often done in the laboratory with synthetic situations and the most apt non-human subjects. "Real world" subjects or settings are maximally similar along some dimension assumed to be important in an effort to establish generality of the obtained functional relationships. Unfortunately they typically introduce many confounds which seriously undercut the actually obtained generality, as well as limit the information they produce. The raison d'etre of synthetic subjects or settings is to maximize our ability to understand the true causal factors and thereby maximize the generality of the obtained functional relationships. Unfortunately synthetic subjects and settings can fail to produce generalizable information due to misunderstood or unaccounted for differences between the model and target.

If the task is simply to develop knowledge applicable to a narrow range of situations such as for a simple technical knowledge, then naive notions of similarity will often adequately serve as a basis for predicting generality. This is the rationale underlying vocational training programs. It is important to note however that if the person is expected to predict in a variety of situations, errors of prediction will occur, and those errors could have important consequences. That is, of course, what separates professionals from technicians. It is the well accepted difference between nurses and physicians. Vocational training or practical experience need not prepare people to generalize into different situations while professional education must.

Rarely will you be content to describe functional relationships applicable to only a few subjects that you happened to select. Typically you will want to make statements which apply to some larger group of subjects. There are two rationales which can be used to select subjects to accomplish that end. A representative subset of the target population can be chosen. This sample is then used to predict functional relationships in the population by virtue of its deliberate similarity. Alternatively, the population values can be predicted based on data obtained from a synthetic model. The prediction in this case is based on the functional similarity between the model and target.

Synthetic subjects are often chosen to allow the maximum degree of control over confounds and to be maximally revealing of the phenomenon of interest. Generality is based on understanding what is going on. Additional important considerations include the convenience of using a particular synthetic subject. A rat is simply easier to deal with than a whale or a person. The optimal synthetic subject is also one which is optimized to reveal information on the dimension of interest, e.g., giant axons of squid. This is especially true in pure research settings where the goal is understanding general mechanisms of action rather than developing an analog of a specific applied problem.

Alternatively, representative samples can be used. If you measure each of a small population from which you determine a functional relationship, you have therefore measured the entire target population and your finding, will obviously apply to its source. However if you are interested in the effects of some variable on the entire population of humans on earth you would have to measure the entire population or measure a sample and then infer information about the population. To be meaningful, the sample must provide a relationship which generalizes to the entire target population, (i.e., all humans). The sampling techniques which you use are very important in that they can dramatically affect the generalizability of your finding. In order to be useful your sample must be similar to the target population or "representative."

You may take advantage of the properties of chance to assure yourself of generality by randomly sampling from the population. Random sampling guarantees (other things being equal), that the obtained functional relationships are accurate because a random sample will produce as many deviation above as below the true value on all dimension simultaneously. If you randomly sample enough individuals you will get an accurate picture of the population and your findings will generalize. If your population or sample is small special precautions are necessary to maintain representativeness. For example, the probability of obtaining an individual can change as the population is depleted if subjects are not replaced.

An alternate sampling procedure is to sample the population such that you deliberately match the sample to the population according to the relevant factors you choose, such as proportionate numbers from each strata. This procedure is used when you cannot obtain a large enough random sample to make it representative. Rational sampling requires that you know in advance the relevant properties of the population and that your sampling procedure guarantees representativeness.

Apparatus or setting variables can strongly affect the generality or the applicability of the functional relationships derived in one situation and applied to another. The selection trade-offs are the same as those involved in selecting subjects. Maximum similarity versus maximum control. “Real world” situations maximize generality through similarity, but also typically maximize error through lack of control. Additionally they are often not optimized to provide the data needed to understand the phenomenon. Synthetic situations on the other hand minimize error through the deliberate choice or design of the apparatus or setting. However, they are often specifically chosen for their ability to reveal fundamental relationships rather than to identify important controlling variables between the testing situation and some specific applied settings.

If a process inferred by our theoretical structure such as fear is in fact a correct description then the procedures we use must accurately reflect that inferred theoretical process. If shock is presumed to cause fear then it must do that and all procedures which are assumed to produce fear must produce the same functional relationships. Otherwise a finding demonstrated with one procedure may not generalize to a situation where the inferred variable is produced or measured in some other way. If verbal behavior is used to index actual behavior then the data must generalize to actual situations. If someone says that they will help in a crisis and we use that fact in our theory then they must in fact help in a crisis. In addition if the experimenter establishes demand characteristics or other confounds which do not exist in the target procedure then the findings may not generalize. The selection dilemma for procedural variations is the same as with subjects and apparatus. Similarity versus control.