Evaluation and Introduction to Science

Introduction to Public Policy

POSC 315 :: Week 12-2

Policy Evaluation

  • Evaluation is research conducted to investigate a social program — a public policy.

  • Big Question: Was there a significant and positive impact to the target group(s) that would have not occurred without the program?

  • Who Cares? Stakeholders: Taxpayers, Program Beneficiaries, Program Administrators, Elected Officials, etc.

Evaluation and Systems Thinking

  • Inputs: Resources used to implement the program
  • Activities: What the program does
  • Outputs: The direct products of the program
  • Outcomes: The results of the program
  • Impacts: The long-term effects of the program
  • Feedback: Information about the program that is used to improve it

Evaluation Designs

Before-and-After Design

  • The policy evaluated for the changes it has produced since its implementation, controlling for other possible influences on the outcome

After-Only Design

  • The policy evaluated for the changes it has produced since its implementation, without controlling for other possible influences on the outcome

With-and-Without Design

  • The policy is evaluated for producing changes in a target population, compared to another population that did not receive the policy, i.e., treatment and control groups.

Time-Series Design

  • The policy is evaluated the changes produced by the policy over a lengthy period; gives us the best information with which to evaluate a policy

Evaluation Questions

  1. Is the program needed?
  2. Can the program be evaluated?
  3. How does the program operate?
  4. What is the program’s impact?
  5. How efficient, equitable, and effective is the program?

“Deeper” Evaluation Questions

  1. What is the program’s impact on the target population?
  2. Was the correct problem identified? Was the program designed to address the problem?
  3. Was the program implemented as designed?
  4. What data were used to evaluate the program? What was left out?
  5. Is there need for a new program? Is there a better way to address the problem?
  6. What should be done next to improve the program?

Public Policy and Science

How do we know what we know?

Science

  • Science is a method of inquiry that seeks to describe and explain the natural and social worlds, and to predict events in the natural and social worlds.

  • Scientific Method is a process of inquiry that uses a set of procedures to systematically test a hypothesis.

Philosophy of Science

  • The analysis and evaluation of the basic concepts and practices within and about science.

  • It deals with the assumptions that undergird the scientific method.

  • It is concerned with the nature of scientific knowledge and the justification for scientific claims.

Three Major Fields Inform Inquiry

  • Metaphysics is the study of the nature of reality.

  • Epistemology is the study of knowledge and justified belief.

  • Axiology is the study of the nature of values.

Metaphysics

The Study of Reality

  • Deals with questions like “What is real?” and “What is the nature of reality?”

  • Ontology is the study of the nature of being.

  • Cosmology is the study of the nature of the universe.

  • Theology is the study of the nature of God.

Epistemology

The Study of Knowledge

  • Deals with questions like “What is knowledge?” and “How do we know what we know?”
  • Rationalism is the view that knowledge is acquired through reason.
  • Empiricism is the view that knowledge is acquired through experience.
  • Skepticism is the view that we cannot know anything for certain.

Axiology

The Study of Values

  • Deals with questions like “What is good?” and “What is right?”
  • Ethics is the study of right and wrong.
  • Aesthetics is the study of beauty.
  • Political Philosophy is the study of the nature of the state and the justification for political authority.
  • Social Philosophy is the study of the nature of society and the justification for social authority.

Methodological Approaches

How should we go about knowing what we know?

Positivism

  • Positivism is the view that the scientific method is the best way to acquire knowledge.

  • Reality exists independently of our minds and is driven by laws of cause and effect which we can know.

  • Inquiry can be free of bias and values.

  • Hypotheses can be tested and verified.

Post-Positivism

  • Post-Positivism is the view that the scientific method is the best way to acquire knowledge, but that it is not free of bias and values.

  • Reality exists independently of our minds and is driven by laws of cause and effect, but it cannot be fully understood.

  • There is a multiplicity of perspectives and interpretations about cause and effect.

  • Hypotheses can be tested and falsified.

  • Paradigm Shifts occur when a new theory replaces an old theory.

Constructivism

  • Constructivism is the view that the scientific method is not the best way to acquire knowledge.

  • Reality exists as a mental construct and is relative to the individual.

  • Knowledge and the knower are part of the same subjectivity entity.

  • Identifies, compares, and describes the various constructions that exist, both hermeneutically (through interpretation of texts) and dialectically (through the interaction of ideas)

Critical Theory

  • Critical Theory is the view that the scientific method is not the best way to acquire knowledge.

  • Reality exists but cannot be fully understood or explained.

  • Social problems stem more from social structures and cultural assumptions than from individuals.

  • Knowledge is a social construct that is relative to the individual and the social context.

  • Knowledge is used to justify power and privilege, to oppress and exploit, and to liberate and empower.

Symbolic Interactionism

  • Symbolic Interactionism is the view that the scientific method is not the best way to acquire knowledge.

  • Humans act based on the meanings that things have for them.

  • Meanings are modified and handled through interpretative processes used by individuals dealing with the signs and symbols they encounter.

  • The meanings of things are derived from, and arise out of, social interaction.

Why does this all matter?

How does this relate to public policy?

Public Policy: Definitions

  • Heclo (1975, 305): Policymaking can be viewed as “collective puzzlement on society’s behalf; it entails both deciding and knowing.”

Public Policy: Definitions

“A definition of the problem is part of the problem: Politics arises because we do not share perceptions of what the problems are, or if we do, what follows from the definition in terms of what can be or should be done… Ill-structured as they are, public issues are not demarcated: we do not know where one problem begins, and another ends” (Parsons 1995, 88-9).

Public Policy in a Democracy

James A. Jones (1974, 561): “Whosoever initially identifies a social problem shapes the initial terms in which it will be debated and the initial range of alternatives that will be considered. The initial identification of a social problem is thus a crucial act of social policy.”

Public Policy in a Democracy

Social construction of problems is not a problem; it’s democracy! Conflict is only a perception. The problem arises when the conflict is not allowed to become something else.

The End

Have a great break!

References

  • Parsons, Wayne. 1995. Public Policy: An Introduction to the Theory and Practice of Policy Analysis. Cheltenham, UK: Edward Elgar Publishing.
  • Jones, James A. 1974. “What’s in a Name? Some Reflections on the Sociology of Diagnosis.” In The Relevance of Sociology, edited by James A. Davis, 560–78. New York: Wiley.
  • Heclo, Hugh. 1975. “Review Article: The Problem of Policy Problems.” Journal of Public Policy 5 (3): 305–18.