Which law schools best prepare students for the bar exam?

Relationship between bar passage and both undergraduate GPA and LSAT scores

Prior to modeling, however, we’ll examine the individual relationships between the predictors and bar passage. This will provide us initial confirmation that we should at least test them in models. Figure 1 shows the relationship between both median undergrad GPA (top plot) and median LSAT score (bottom plot) and bar passage. Each reveal a positive relationship between the admissions factor and bar passage.

Figure 1: The top plot shows the relationship between a school’s median undergrad GPA and bar passage rate. The bottom plot highlights the relationship between median LSAT and bar passage. Both median undergrad GPA and median LSAT correlate with bar passage.

Correlation between median LSAT and median undergrad GPA

We’ll next look at the correlation between median undergrad GPA and median LSAT. Our prior is that these two predictors are highly correlated, as more elite schools have higher median undergrad GPAs and LSAT scores. This could cause colinearity problems within our models, which might widen the uncertainty of predictions.

Figure 2 confirms our prior: median undergrad GPA and median LSAT are highly correlated. In the plot, both predictors are standardized to have a mean of 0 and standard deviation of 1, allowing us to compare their linear relationship - the blue line - with a prefect correlation represented by a line with a slope of 1 - the red dashed line. The plot shows an extremely strong relationship between median undergrad GPA and median LSAT. The linear relationship is only slightly flatter than the perfect relationship (red dashed line).

Figure 2: Correlation between each school’s standardized median LSAT and standardized median undergrad GPA. Blue line is the fit line of the data, red line is a hypothetical perfect fit line - a slope of 1. LSAT and GPA are highly correlated, with the actual fit line mirroring the perfect fit line.

Association between 1L attrition and both median LSAT and median undergrad GPA

Let’s also look at the association between first year (1L) attrition and both median undergrad GPA and median LSAT. Our expectation, based on prior knowledge, is that schools with lower median undergrad GPA and LSAT scores will also have higher attrition. Figure 3 confirms this prior. But, the relationship is not too strong.

Figure 3: The top plot shows the relationship between a school’s median undergrad GPA and 1L attrition. The bottom plot shows the relationship between median LSAT and 1L attrition. Both median undergrad GPA and median LSAT correlate slightly with 1L attrition.

Association between 1L attrition and bar passage

Finally, figure 4 highlights the association between attrition and bar passage. Not surprisingly, it’s negative: schools with higher attrition rates have lower bar passage rates. This finding is not unexpected because we saw from figure 1 that higher median undergrad GPA and LSAT scores correlate with higher bar passage rates and figure 3 showed us that lower median undergrad GPA and LSAT scores are associated with higher attrition rates. Putting both findings together, we would assume that higher attrition rates correlate with lower bar passage rates.

Figure 4: Correlation between bar passage rates and 1L attrition. There is a slight negative relationship, but there is also a lot of variance in the relationship.

Models with admissions factors as predictors

We now turn to the heart of this post. We’ll predict each school’s bar passage rate and see how actual rates stack up against predictions. But, we need a good model to do this. We know our potential predictors: median, 25th percentile, and 75th percentile undergraduate GPA and LSAT score, and first year attrition. But, we don’t know the functional form of the model that provides the best predictions. Do we need all predictors or only a couple of them?

We will use two steps to search for the best model. First, we will conduct model comparison and validation on four different models that only include the undergrad GPA and LSAT predictors, along with year and state. Then, we will take this model and compare it against models with attrition as an additional predictor. Models will be compared by testing their predictions out of sample using leave-one-out cross validation (LOO). Posterior prediction checks will also be used to ensure the fit is reasonable.

All models are Bayesian hierarchical logistic regressions. The response is a school’s bar passage rate for a given state and year, with the model weighted by the number of takers. All models have the same group-level predictors: state and year. They only differ in their school-level predictors. The four initial models are below:

1. Linear: median undergraduate GPA and median LSAT score for the given graduating class year.

2. PCA: Figure 1 showed that median undergraduate GPA and median LSAT score are highly correlated, raising fears of colinearity. The 25th, median, and 75th percentiles of each metric are also highly correlated with themselves (the 25th percentile undergrad GPA is correlated with the median undergrad GPA). We’ll use Principal Component Analysis (PCA) to reduce the six highly correlated percentile variables into a smaller number of uncorrelated ones, while maintaining as much of the information in the variables as possible. The table below shows that the first two principal components contain 98% of the variance in the six LSAT and GPA variables. By only using these two principal components as predictors, we get 98% of the variance with only two uncorrelated variables.

3. PCA spline: To account for possible non-linear relationships between the two PCA variables and bar passage, we’ll examine a model with splines added to the two PCA variables.

4. Spline: This model contains median undergraduate GPA and median LSAT score like model 1, but adds splines to them.

Compare admissions factors models using LOO

The table below compares each model by its expected log posterior density (ELPD). The Difference in ELPD column shows the difference in ELPD between the best fitting model - the model on the first row - and the model in the given row. The PCA spline model performs best. Plus, the standard error of the difference in ELPD between the best and second best model is small enough that we can have confidence that the PCA spline model did not perform best due to some randomness in the sampling draws. Assuming the model’s posterior predictive check pans out, we’ll start with it when adding attrition.

Poserior predictive checks of admissions factors models

Posterior predictive checks (PPC) simulate y values - bar passage rates - from the models. Our checks simulate 200 distributions of bar passage rates based on 200 model simulations. We can then compare these distributions to the actual distribution of rates. Ideally, they align.

Figure 5 shows the PPC plots. All models have the same problem: they overpredict at the 75% to 90% bar passage rates and underpredict at the tails. This is shown by the dark line (actual distribution) being lower than the light lines (simulated distributions) at the 75% to 90% range and higher than the light lines outside this point. The takeaway from the misfit models is that there will be too many predictions in the 75% to 90% range and not enough at each tail.

Figure 5: Posterior predictive check for all models. A problem in all models is that they fail to accurately predict at the most likely bar passage rates. We know this because at the highest points in the curve, the predicted passage rates for schools is higher than the actual rates.

Add attrition as a predictor

Step two involves using the best model in the previous section, PCA with a spline, and adding attrition as a predictor. We’ll incorporate attrition using the natural logarithm of attrition. The natural logarithm is used because attrition rates have a right skew: most schools have attrition rates between 0% and 15%, but some schools have rates greater than 30%. Using the natural logarithm removes much of the right skew and makes the distribution more symmetrical.

We’ll create two different models with attrition, each an extension of the PCA spline model. The first adds the natural logarithm of attrition, centered, as an individual predictor. The second also includes the natural logarithm of attrition, but wraps it in a spline to model non-linearity.

The table below compares each model based on the ELPD using leave-one-out cross validation. We see little difference between the two models with attrition added, but these two models are better than the model without attrition.

Moving on to the model check, the plot of posterior predictive checks in figure 6 shows that the models with attrition have the same problem we encountered in figure 5. More predicted pass rates fall between 70% and 90% than would be expected given the distribution of actual bar passage rates.

Despite the non-ideal fit, the PCA spline model with attrition will be used to predict bar passage rates of schools. This model performs almost identically to the PCA spline model with attrition added as a spline, but is simpler. All things equal, we’ll opt for the simpler.

Figure 6: Posterior predictive checks. The attrition model cehcks reveal the same problem as the models without attrition - too many predictions in the 75% to 90% range.