How Colleges Measure and Prove Their Value

Colleges are under growing pressure to prove their value to students, parents, legislators and others. The scrutiny can be uncomfortable, but more are responding with serious efforts to measure and explain their value.

A recent episode of The Key, Inside Higher Ed’s news and analysis podcast, examined the data and metrics colleges and universities are using—or should explore going forward—to gauge the value they provide to their students and other constituents.

Three guests joined the conversations: Michael Itzkowitz, senior fellow in higher education at the center-left think tank Third Way; José Luis Cruz Rivera, president of Northern Arizona University and a member of the Postsecondary Value Commission; and in a separate discussion, Pamela Brown, vice president for institutional research and academic planning for the University of California president’s office.

An edited transcript of the discussions follows.

Inside Higher Ed: Michael, you were on The Key last spring talking about the price earnings premium that you and Third Way had developed to measure value in higher education. Can you give us a quick overview of how you’ve seen those data and other efforts to measure postsecondary value influence both the public conversations about higher education and, to the extent you’ve seen it, the behavior of colleges and students?

Itzkowitz: Over the past year, we’ve seen an uptick in students and families wanting to know what type of return on investment they’re going to get from the college education. In 2020, we released a paper about a new way of evaluating institutions of higher education. We were looking at the cost students pay out of pocket to obtain a specific credential, relative to the earnings boost or premium they obtained by attending a specific college.

We define earnings boost as the amount more that they are making beyond the typical high school graduate within the state that their institution is located. When we have the earnings boost relative to the cost, we can get a better understanding of how long it takes students to actually recoup their educational investment after they attend a specific institution. We’ve seen the broader higher education community latching on to this narrative just to get a better understanding of whether or not specific institutions and programs are paying off. We have data from the U.S. Department of Education that allows us to look at, if you study engineering or social work or psychology at a specific institution, what’s the ROI you’re getting, looking at your cost relative to the earnings boost you’re obtaining?

We’ve seen students and families using this information, but we’ve also heard from institutions starting to understand that the data is there and showing an interest in the program-level data in terms of how they improve their programs and all of their offerings writ large.

Inside Higher Ed: José Luis, you were a member of the Gates Foundation’s Postsecondary Value Commission when you were president at Lehman College, and since you’ve been president of Northern Arizona you’ve made delivering equitable “postsecondary value” a key part of the university’s new strategic vision. Can you tell us why you’ve made that a focus, how you’re defining that term, and how you’re measuring that value?

Cruz Rivera: Northern Arizona University has a long legacy of providing opportunity to higher education within Arizona and beyond. As we were starting our 17th presidency, we had an opportunity to have a universitywide conversation about how to ensure that the excellent education we provide was available to more students. It became clear that the concept of equitable postsecondary value would serve as a good north star. Our newly approved vision as a university, part of our NAU Charter, calls on us to become the nation’s pre-eminent engine of opportunity, vehicle of economic mobility and driver of social impact by delivering equitable postsecondary value to the people of Arizona and beyond.

We need to ensure that more Arizonans have access to an excellent postsecondary education, that the quality of our programs and the way they are designed and aligned with workforce needs will provide them economic mobility, and that we do not lose sight of the fact that—in addition to the individual or private gain that our students as a graduate will have from their degree—we’re also very much about driving social impact. The way we go about educating our students and the careers that we educate them for has to be accounted for, as well as value to the people of the state.

Inside Higher Ed: How do you define value?

Cruz Rivera: We’re looking at everything from our ability to expand access in Arizona, the enrollment and the types of students that we’re admitting, all the way through persistence and graduation, and then postcollege outcomes, their labor wage data. We are looking at the Third Way index and other economic mobility indices, like the IHEP equitable postsecondary value explorer. We’re also looking at our total number of graduates in areas like social work, teachers, early childhood caregivers, mental health specialists and nurses.

Inside Higher Ed: One of the interesting questions around this is how quantitatively we should be able to measure certain things. Higher education has never been terrifically good at measuring its impact or … measuring learning. Career outcomes are measurable, and we’ve seen a lot of movement by governments and organizations like Michael’s in driving those data forward.

We are pretty dependent on postcollege earnings to define value. That makes sense given that that’s a primary, if not the primary, reason many people get post–high school education and training. But should career outcomes remain the dominant definer of value?

Itzkowitz: Depends which perspective you’re looking at. I’m specifically focused on federal policy; I want to help ensure that federal taxpayer dollars are being used effectively and efficiently and that students have at least minimally good outcomes to where they’re not worse off after they attend a federally funded institution or college program. That bar is so low, it’s almost nonexistent in terms of how we measure college programs or institutions, and a very reasonable starting point.

There are 6,000 college programs across the United States where graduates, two years after they leave, are still earning less than the typical high school graduate in the state where their college is located. This ranges as low as $22,000 in some states, and not much higher in many more. Think about paying this money, through personal finances or loans, earning your credential or degree, and you’re still making below that minimal threshold. If you ask any student or any family whether they would be comfortable taking out loans and taxpayer grants and personal finances to have the sort of outcome, I almost guarantee 98 percent would say no.

This minimal threshold … helps you identify the most risky programs. Once you get past those programs, it becomes more of a question of how do we address a college’s effectiveness … and whether or not they’re actually producing the outcomes that we want in higher education.

Inside Higher Ed: Once data are out there, it’s very difficult to limit, control, influence how they’re used. Where I think it gets tricky is when you start seeing politicians say, “Well, why are we supporting social work programs?” It is all about how it’s used. Given the bar that you just described, what else do we do to enhance and flesh out the discussion about college quality and college value?

Itzkowitz: You put the data out there. Everyone has access to this; you can use it in a bunch of different ways. If you’re a college administrator, you know about this data, or you should be aware now. If I’m a community college president or college president, I have data on all 50 of my programs. I can see that 40 of them are working really well, five are on the fringe in terms of economic outcomes and five are leading the majority of my graduates to not even earn as much as a typical high school graduate where my institution is located.

This is a good starting point to ask questions: Is my program not working the way I think it is? Is my curriculum not matching up with the skills that are required in terms of being successful economically within the field that we’re preparing students to enter? Is there something going on with my career service center so we’re not matching graduates as effectively as we should be? Or are we offering credentials where there aren’t jobs within the geographic region where we’re offering them? This is actionable information that allows administrators to constantly iterate, to start asking these questions and continuously improve their institutions as a whole.

Inside Higher Ed: Luis, you’re starting to develop and collect data and metrics that you want NAU to be judged by. What else is in that mix? And how do you think about career outcomes as part of the larger whole?

Cruz Rivera: So there’s various aspects to that. We want to look at civic engagement, voter behavior, cultural and artistic contributions, contributions to innovation and science, basic and applied. There are various metrics that will balance, if you will, the overall contribution that NAU makes through its graduates to the state. Focusing solely on what the data would allow us to do as leaders of institutions, and I think Michael presented a good way of thinking about it, is an entry point into better understanding how your portfolio of academic programs is faring, not only for individuals, but also to the communities they represent.

Here’s one example. If we look at our portfolio of programs right now and try to think about the quality or the importance of those programs, purely from the perspective of premium earnings of our graduates, our teachers’ program would not be faring very well, because we’re in a state that ranks 50th out of 50 in teacher pay. We’re going to shut down our teacher education program. It calls on us to think more strategically about the cost of those programs. That is something that NAU and Arizona as a state have been working on for the last several years. We have something called the Arizona Teachers Academy, which will provide students a tuition-free teacher-prep degree, provided they commit to staying in the state and teaching in a school in the state for the same number of years that they get this scholarship.

It’s less about what are the low-performing programs on this metric and more about how do we ensure that this is a viable program from an economic perspective to the individuals, knowing that we do need excellent teachers out in our schools.

Inside Higher Ed: Some people in higher ed bristle at these discussions because of this sense that we’re oversimplifying [value] and judging it on too narrow a set of metrics. Some don’t like to be asked hard questions. We’re in an era of evidence from which we’re not going back. Do you think some of the noneconomic factors are calculable? Do you think it’s incumbent on institutions not just to say, we encourage philanthropy or more voting or better health outcomes, but to prove it, which I think a lot of institutions have been disinclined or not prepared to do?

Cruz Rivera: There are ways to capture that. At NAU we appointed for the first time a chief economics adviser to the president who is designing some studies that will allow us to get some of those noneconomic value metrics, from the standpoint of what Arizona voters and taxpayers would be willing to invest in, in order to get those noneconomic outputs available to the state.

We need to remember as we think about public institutions, particularly four-year public institutions, that we’re called upon to do many things: expand access, increase retention rates, improve learning, increase graduation rates, prepare graduates to be workforce ready, push the frontiers of knowledge, create more inclusive environments, economic development in our communities. We need to make sure that that portfolio of metrics captures everything that we are entrusted to do on behalf of a state.

Inside Higher Ed: Michael, I don’t see the federal government ever trying to come up with a way of measuring overall contributions by institutions to society. That’s not really its job. Do you see the accountability part of this pretty much stopping at the economic outcome level? Or are there other ways governments should be thinking about the institution’s contributions?

Itzkowitz: Most of us engaging in this topic have probably attended a four-year school, and we have this idea of what that means. We’re thinking about students hanging out on the quad, engaging in really cool ideas, having Hare Krishna lunches and critically thinking in all of the classes they may take. This comes from a sociology major right here.

A lot of institutions are less than four-year institutions, and they are specifically focused on offering career-oriented programming that leads students to acquire a specific skill set and enter the workforce immediately. We can see through data that disproportionately these shorter-term programs are oftentimes the most likely to have economic outcomes that raise a flag we can see as troublesome, and as I define it as not meeting its very minimal economic threshold.

It’s very difficult to ask institutions to think about some sort of national reporting structure that gauges them on civic engagement, being involved in a local community, being an active citizen and contributing to society, the way that a lot of us think they should and that we value. Think about the teacher example that José Luis brought up. People often worry about liberal arts majors in these conversations. Liberal arts majors have some sort of ROI, and while they might not have it immediately in terms of how much they’re making, we can see an exponential increase in their salary after the first few years. The worries for some of these four-year programs aren’t as big as they’re presented.

But we need to think broader in terms of accountability. A lot of programs are not four-year programs, they are career-oriented programs. And that’s really important in terms of accountability writ large.

Inside Higher Ed: We had an accountability regime put in place during the Obama administration, the gainful-employment rule, which applied only to the most directly career-relevant programs. There’s a fair bit of discussion about whether there ought to be “gainful for all.” Kevin Carey just proposed gainful for graduate programs. Do you think it would make sense to have some kind of floor, even for the non–purely vocational programs?

Itzkowitz: We need to just look at the data and understand the facts. Students, whether they attend a public or private or a for-profit institution, their No. 1 concern is for greater employability and to have a financially secure future. I don’t think that necessarily just applies to students in career-oriented programs.

A few years back, [former U.S. senator] Lamar Alexander did propose something called “gainful for all,” based off loan-repayment rates. Our loan system is all over the place right now due to the recent announcement on forgiveness. That’s another reason to look at employability as a floor. Most programs and institutions are going to hit this floor without a problem. The gainful-employment regulation specifically is to cover all for-profit programs and career-oriented programs at public and private nonprofit institutions. There is some discussion of how we can implement this floor for all institutions. The [Education Department] is strapped right now in terms of statute and what they can and what they can’t do. But they are interested in putting out this information to where I and others can use it to spur conversation.

Inside Higher Ed: José Luis, how would you react, and how do you think some of your colleagues might react, to a proposal that would ask all programs to meet the kind of floor that Michael’s talking about?

Cruz Rivera: Accountability is important. The details are also important, to see how accountability is being defined and how nuanced it is, how much space it provides institutions to contextualize any areas in need of improvement. I welcome the conversation.

In the meantime, there’s a lot that our accreditation agencies can do without hard floors to encourage the continuous improvement that is needed, across particular programs, within a broader portfolio in a particular university, where you have certain professions that perhaps the earnings are not commensurate with the cost of acquiring that degree. There is a lot of introspection we in higher ed need to do.

Inside Higher Ed: One possibility between government regulation and a Wild West is greater agreement among institutions themselves about how they should be judged. We’ve seen higher education come under increasing scrutiny, I think, partly because the industry hasn’t been clear in explaining how it wants to be judged. Would there be benefit to trying to get a coalition of institutions to publish common information and judge themselves in somewhat the same way?

Cruz Rivera: The American Association of State Colleges and Universities has put together a group of about 10 institutions that are working through the implications of the equitable value movement and the data that is available to our institutions, to think through how we can come together and present a common way of presenting ourselves for accountability. Over the next year, with the funding of and backing of the Gates Foundation, there will be new ways of thinking about this.

Inside Higher Ed: Michael, beyond whatever you think should happen from a federal regulatory perspective, do you think there’s opportunity for governments and other organizations to influence how colleges behave?

Itzkowitz: Twenty thirteen was the initial College Scorecard that was put out. President Obama at the time made a speech that said, “We might actually use this data to do something like rate institutions.” And for the next two years, institutions let everyone know that they were not a fan of this. If they want to be rated, they want to be rated on what they’re best at. The first reaction [when you engage in this conversation] is “absolutely not,” and then it’s, “OK, the data is there, we’re going to be looked at regardless of what we think.”

The data is at a point now of “how do we use this for continuous improvement?” From a national perspective or from the college perspective, a lot of folks in higher education are there for a reason, and it’s because they want better outcomes for students.

---

Here’s an interview with Pamela Brown, vice president for institutional research and academic planning at the University of California president’s office.

Inside Higher Ed: The UC president’s office recently undertook a major project to define and show the value of a degree from the university. Can you tell us briefly why the office undertook that and a couple of the most important takeaways?

Brown: We had gotten some data from the State Employment Development Department, which allowed us to get a better understanding of postgraduate earnings for our alumni. It was a way for us to take a look at things like debt-to-earnings ratios and what happens after [students leave]. There were a number of questions that we could answer. But we realized that that view was narrow and that alumni outcomes are much greater.

So my team took a look at different frameworks that were out there. We landed on something called the Post-Collegiate Outcomes Initiative, which a number of college associations, including the Association of Public [and] Land-grant Universities, pulled together. It gave you a framework to look at public and private benefits along an economic and well-being measure. It gave us a much broader appreciation of the importance of a college degree and what it means.

One thing that stood out to me was the importance of economic mobility and how earnings continued to rise after students graduated from the University of California. In particular, when you looked at Pell Grant recipients, you’d see this intergenerational mobility where five years after graduation, they’d earn more than their families did as a whole. For first-generation students, it was seven years.

The other thing that really stood out was the important role that our graduates play in job creation. Getting a UC degree didn’t just mean that you got yourself a job, but oftentimes, you were creating jobs that would then employ others. There was this interesting combination of data, and then stories that could illustrate the importance of it to a wide range of audiences—the public, legislators, our board. It could be parents and students themselves.

Inside Higher Ed: That’s where I wanted to go next: Who is this for? What kind of pressure, and from whom, is an institution like the University of California feeling … to prove itself?

Brown: Many audiences. The question kept coming up, and we knew it would continue to come up. We thought that instead of approaching it in an ad hoc way, it would be good for us to do research to [help our campuses] respond to questions along these lines. The report that we produced, “What Is the Value of a UC Degree?” is to support them in being able to communicate out the value but also to support other colleges and universities looking to do the same thing.

There’s no question to me after having done this research about the really important value of getting a college degree and particularly with the University of California, there are a number of ways where you see the benefits that alumni have, there are other places where it highlights if we do more work and support, we can better improve those outcomes. In part based on this research, we’re prioritizing reducing the time to get a degree, particularly for what we’re calling “new-generation students”—Pell, first-generation and underrepresented students that are the majority of the public high school students in California.

Our data showed that students who graduate in six years instead of four left with $7,500 more debt, and over 16 years of working, their earnings were $200,000 less. So we’re focused on efforts to provide more support for timely graduation.

Inside Higher Ed: More broadly, do you think that the questioning about value is fair, reasonable, understandable? You talked early on about your sense that the economic outcome prism was too narrow to capture it all. What are the key things institutions should be doing for themselves to assure they’re providing the value that they purport to? And for their constituents?

Brown: There are basic things to look at: Do your students graduate? How long does that take? Do they leave with debt? How much is that? Are they able to get employed? Is that employment sufficient to be able to repay the debt and move up to the middle class if not beyond?

Some of those simple measures can provide a baseline to start with. But we looked into things that aren’t as easy to quantify and are more illustrative. We got access to LinkedIn data to try to understand occupation. We can get an understanding of industries and other things from public sources, but what specific jobs alumni have is more difficult.

We did a very muddy project to pull that information together. We looked at undergraduate alumni, graduate alumni, and we could start to talk about the kinds of jobs they had. And we could talk about the kinds of roles that students in arts and humanities played, which often were leadership positions, either management or C-suite offices. That gave us a different way to think about it. Oftentimes you think STEM means tech. But tech has a lot of different kinds of students that they need, with different backgrounds.

The other thing is that often the measures we have are just after graduation. As we’re continuing to collect data, we see the trajectory of incomes continue to rise. That’s really important. It happens across all disciplines between two and 10 years, earnings double. As we continue to go on, we see that they merge closer together.

Inside Higher Ed: How do you think about and protect from quick judgment programs that are designed to lead to fields that are notoriously low-paying, like social work and education? There are fields that are hardly vocational, yet instead of producing people for computer science, they produce people for fields that we don’t as a society value through pay as high as some others’. How do you think about the importance of this task for those fields?

Brown: It’s about communicating what the value is. STEM majors benefit from having training in arts and humanities fields; they need to be able to communicate, to work with others, and they often take courses in those majors to broaden their experience. No matter what program you’re in, you are likely having some exposure to those fields. Then if you think tech, we’ve got numbers of examples of how arts and humanities and social science majors are benefiting the technology industry that’s there.

And especially when we’re talking to state legislators, their background is in those fields, the social sciences and arts and humanities. Now we’ve got decades of earnings data that show a continued increase for those in arts and humanities. Not only can you have a stable life, but you can do quite well.

Inside Higher Ed: Having gone through this exercise, which I imagine will never quite be finished, do you come away feeling more confident than you might have been 10 years ago, when most of what we were thinking about the value of a degree was in our hearts and our guts? Do you think that the framework you’re building makes a strong case and enhances the way people might think about the value of a UC degree?

Brown: I definitely do. We’re able to answer some of the basic questions that people have about what happens: Will I earn enough to be able to pay for my college and to have a good life? The other thing we can do is demonstrate why it’s important to have an educated citizenry. We can show that there’s increased volunteers, and an increased likelihood of people that are voting, participating in public service roles that are critical to society. It’s been a really interesting project. And you’re right, it’s not done. We’ll continue to add on to it, to learn from other institutions about the approaches they’re taking, and hopefully be able to demonstrate where we can do better as institutions and why education is really important for all of us, not just individuals, but society as a whole.