Improving teaching and learning at all levels is a national priority. Academic disengagement of law school students, especially third year students, has raised concern among legal educators. As graduates begin their careers, many may not be as well prepared as they need to be.

Accreditation processes serve as one means of evaluating the law school experience. However, such reviews are infrequent and do not require schools to identify many of the specific educational processes that lead to valued outcomes. Information about student behaviors and school environments that contribute to the outcomes is essential in order to know where to focus attention and resources in ways that will enhance student learning and law school effectiveness.

One initiative that serves this purpose at the undergraduate level is the National Survey of Student Engagement (NSSE). Based at the Indiana University Center for Research, Policy and Planning, the NSSE project assesses the extent to which undergraduate students engage in educational practices associated with high levels of learning and development. More than 730 different colleges and universities and 435,000 students nationwide have participated in the project in its first four years.

Building on NSSE's experience, visibility, and wide-spread use, we introduce a student engagement survey for law school students. The Law School Survey of Student Engagement (LSSSE - pronounced "lessie") will provide law school faculty members and administrators with information that can be used almost immediately to identify where improvement can be made in the local context.

This past spring, we pilot tested the LSSSE with approximately 4,300 students from eleven law schools across the country. LSSSE surveys were administered online from March to June 2003. In an ongoing process, the project is incorporating feedback from students, faculty members, law school administrators, and others during the pilot test to prepare for a larger national administration in the spring of 2004.

Table 1 provides a brief gender and race profile of the students who responded to the LSSSE 2003 pilot test.

Class

The 4,325 respondents were split fairly evenly between 1L (37%), 2L (31%) and 3L students (30%). Two percent of respondents were 4L students.

Enrollment

Only 6% percent of respondents were part-time students. Three percent of students transferred from another law school.

Age

Ten percent of the respondents were under 22, 47% were 23-25, 30% were 26-30, 10% were 31-40, and 4% were over 40 years old.

Marital Status

One-fourth (26%) of respondents were married.

Before Law School

Students were asked how much time they spent between earning their undergraduate degree and enrolling in law school. Almost two-thirds enrolled either directly (34%) or 1-2 years later (29%). Twenty-two percent enrolled 3-5 years later, 10% indicated 6-10 years, and 5% over 10 years after earning their undergraduate degree.

Grades

Approximately one-half of students reported grades of either A/A- (19%) or B+ (32%), while 30% reported mostly Bs. Twenty percent reported B- (12%), C+ (6%) or lower grades (2%).

Law School Debt

Two-thirds (65%) of students reported that they will have $40,000 or more of educational debt upon law school graduation. Fourteen percent said they will have $20,001-$40,000 of debt. Only 15% percent indicated they will have no educational debt.

The following section provides a selected view of the nature and frequency of law school student engagement in various effective educational practices.

Law School Activities

A major focus of the LSSSE survey is on the nature of activities in which students engage, inside and outside the classroom. "Frequent" activity is defined to be at least 50% of all students reporting "often" or "very often" (Table 2).

The least frequent activities are those where the percentage of students who responded "never" exceeded 35%, meaning that roughly one third or more of the students had no experiences in these areas during the 2002-2003 academic year (Table 2).

Law School Environment

Another way to gain insight into the student experience is to look at the extent to which students perceive that their school is a supportive learning environment.

• More than eight out of ten (84%) students reported their school emphasized spending significant amounts of time studying and on school work to a substantial degree ("very much" or "quite a bit")
• More than two-thirds (71%) of students indicated their school emphasized encouraging the ethical practice of law to a substantial degree.
• Almost one-half (45%) of students said their school did very little to help them cope with their non-school activities (family, work, etc.).
• About one-third (31%) of students said their school did very little to provide the financial counseling they need to afford their education.

Enriching Educational Experiences

The types of educational programs and opportunities in which students take part can add value to their school experience. Table 3 highlights the percent of students who reported having participated in, or planning to participate in, some of these complementary experiences.

Time Usage

What students put into their education determines what they get out of it. The amount of time spent preparing for class, participating in lawschool sponsored activities, or participating in community organizations is positively correlated with self-reported educational and personal growth. Time spent on legal pro bono work not required for a class is positively correlated with student gains in contributing to the welfare of their community. Time spent working for pay, caring for dependents, or commuting is not positively related to any educational and personal growth items on the survey.

• More than half (51%) of all students spend more than 20 hours per week studying. One-fifth of students (20%) spend more than 30 hours per week. However, almost a fifth (18%) spend ten or less hours per week preparing for class.
• Twice as many 1Ls study 20 or more hours a week compared with 3Ls, 67% and 34% respectively. (Figure 1).
• Four out of every ten students (40%) work for pay, although this number varies significantly by class level (Figure 2). About one out of four (23%) students work more than 6 hours per week.
• One-fourth (28%) of students do not participate in school-sponsored activities such as student organizations, journal, moot court, and student government, while one-quarter (25%) spend more than 5 hours per week participating in such activities.
• Roughly one-fourth (29%) of students provide care for dependents living with them.

Student Satisfaction

Students were generally satisfied with their law school experience. Eighty-two percent of all students rated their law school experience "good" or "excellent" (Figure 3). Only 3% said their experience was "poor." More than three-quarters (83%) of students would "probably" or "definitely" attend the same school if they were starting over again.

• Students were most "satisfied" or "very satisfied" with their school's library assistance (94%), computing technology (78%) and financial aid advising (68%).
• Four in ten students were dissatisfied ("unsatisfactory" or "very satisfactory") with job search help (45%), career counseling (44%), academic advising (40%) or personal counseling (40%).

Before sharing your LSSSE results school wide, become familiar with the nature of the data and "story line" of your school's performance. Here are some things to consider.

Check the Representativeness of Your Respondents

Even though the survey was administered online to all J.D. students at each law school, an essential early step in looking at your results is making sure that the student characteristics listed in the various reports are comparable to other school-reported data. For example, most LSSSE 2003 schools have few 4L students, so the results for this group may be skewed by the responses of only a handful of students.

Look Carefully at Items with Large Effect Sizes

In the Means Comparison Report, asterisks (*) mark those questions where your students' responses differ at a statistically significant level from students at other law schools that participated in the LSSSE pilot test. Because of the large number of students surveyed, we noted several statistical significance thresholds to reduce the probability that the differences noted are due to chance (p < .05, p < .01, and p < .001). Even so, the actual magnitude of some item score differences may seem trivial, even though they are highly reliable and statistically significant.

For this reason we also report the effect size associated with those item comparisons that are statistically significant. The effect size represents the magnitude of the difference between the student or school behavior represented by the item and the average of the other law schools in the pilot test. When the effect size is large, or a pattern of small or moderate effect sizes exists, it's likely that the quality of the student experience is appreciably different and, therefore, may be of practical as well as statistical significance in the respective area of student engagement.

Finding large effect sizes is not that common in most areas of nonexperimental educational and social science research. So, if your results include some medium or large effects, something may be going on that warrants attention, especially if other empirical or anecdotal information corroborate the LSSSE data. Here are some general guidelines for determining the relative importance of the Cohen's d effect size:

Look for Patterns in Item Differences

In addition to focusing on items with medium to large effect sizes, look for patterns in your students' responses. For example, are your students consistently above or below the mean of the LSSSE 2003 group in certain areas of engagement? Are the differences explainable, perhaps a function of your law school's location, the institutional mission, or certain student characteristics?

Also, don't rely exclusively on statistical significance tests to identify areas that warrant attention. A consistent pattern of scoring above the mean, even though all the items may not reach statistical significance, may indicate your school is doing the right things in terms of good educational practice. At the same time, some schools have very high expectations for student performance and may fall short of their own aspirations even though comparisons with other LSSSE schools are favorable.

Standards of Comparison

LSSSE data serve a diagnostic function by identifying a school's relative strengths and weaknesses in terms of effective educational practice. Comparisons with aggregated data from the other LSSSE 2003 schools can help reveal aspects of school and student performance not readily available from other sources.

There are two basic approaches that many schools use to reflect on their results. One or both may be appropriate, depending on your institution's situation.

Normative Approach

The normative approach compares your students' responses to those of students at other law schools. In the case of the pilot test, it is the other ten participating law schools. This information is readily available from the Means Comparison Report. Breaking down the information into subgroups or engagement topics may be a way of stimulating faculty interest and discussion in the findings.

Criterion Approach

A second way of looking at your results is a criterion-referenced approach, whereby you compare your law school's performance against a predetermined value or level that you and your colleagues deem appropriate for your students, given your school mission, size, curricular offerings, selectivity, demographics, and so forth.

There are several things to think about as you review your law school results.

1. Make sure faculty members and administrators understand and endorse the concept of student engagement. The value of student engagement results for improving teaching and learning needs to be convincingly explained to those faculty members less familiar with assessment in general and the engagement concept in particular.
2. Understand what student engagement data represent and use the results wisely. It takes time, perspective and experience to understand and make the best use of student engagement results.
3. Report student engagement results in a responsible way. We encourage schools to share their results in ways that lead to a better understanding of the student experience and promote law school improvement efforts.
4. Don't allow the numbers to speak for themselves. Every number and comparison reported should be accompanied by an explanation and interpretation of what can and cannot be concluded from the results.
5. Examine the results from multiple perspectives. Use both normative and criterion-referenced approaches to challenge assumptions about student and school performance in the context of your own law school. It's also wise to compare the engagement levels of specific student groups, such as 1Ls to 3Ls, men to women, different legal specialties, etc.
6. Link the results to other information about the student experience and complementary initiatives. The positive impact of student engagement results will be multiplied if the data can be made relevant to groups of faculty and staff working on different improvement efforts within your law school.

Thank you very much for participating in the pilot test of the Law School Survey of Student Engagement. Testing the instrument gives us an opportunity to fine-tune the survey administration procedures, get feedback from students, faculty, and administrators on how they interpret the results, evaluate the usefulness of this LSSSE report, and chart a course for future modifications.

We are pleased to introduce LSSSE because it promises to provide much-needed information about the law student experience and the educational effectiveness of law schools. We are confident that law schools, professional associations, and students will all benefit from examining the results and discussing strategies for improving teaching and learning.

We welcome your comments and feedback and look forward to working with you in putting your LSSSE results to use.

Law School Survey of Student Engagement

Indiana University
Center for Postsecondary Research
1900 East Tenth Street
Eigenmann Hall Suite 419
Bloomington, IN 47406-7512

Phone: 812-856-5824
Fax: 812-856-5150
E-mail: lssse@indiana.edu
Web: http://www.iub.edu/~nsse/html/law-school.shtml