This is the second of my pieces on straightforward, practical ways that institutional arts funders can be more effective. Here, I want to discuss the challenges of evaluating applications in terms of their artistic merit. Naturally, if you’re going to the trouble of funding the arts, you may as well try to fund the good stuff. As such, most funders rightly place artistic merit near the top of their priorities—even programs not specifically aimed at merit usually have an indirect connection via arts education, underserved niches, emerging artists, etc. Given this focus, materials that demonstrate excellence in art making, such as work samples and press clippings, feature prominently in application guidelines.

But noble intentions aside, artistic merit is difficult to judge. It has a tendency to get overshadowed by more objective but less relevant criteria, often without anyone realizing it. When you set out to pick the best applicants, it’s unfortunately not enough to create fair guidelines and lock a bunch of arts professionals in a room. The squishy nature of artistic merit creates unintended incentives that steer your evaluators toward tangential topics, often undermining your mandate in the process.

Freakonomics authors Stephen Dubner and Steven Levitt provide a rather poignant example of how powerful these types of unintended consequences can be in their latest book. In the 1990s, Mexico City instigated anti-pollution laws, assigning every car a day of the week when it couldn’t be operated. The end result, however, was not the desired decrease in air pollution. In fact, they saw an increase in smog. Why? Because people whose needs weren’t met by transit just bought a second car for use on the off days, and that car was usually an old, fume-belching clunker.

Luckily, unintended consequences in the arts aren’t usually hazardous to health. Still, when it comes to assessing artistic excellence, we see a similar dynamic where the letter of the law often contradicts the spirit. I don’t know how many times I’ve heard jurors say, after results were announced, “Well, I didn’t think Project A was very strong, but I guess the other guys did.” They probably didn’t. Rather, some poor-fit criterion got subbed in for artistic merit, leading assessors to make decisions based on factors that may or may not have anything to do with the quality of the applicants’ work. You get an “emperor’s new clothes” situation where everyone thinks better applicants got turned down but everyone also assumes they’re the only ones who feel that way.

It doesn’t need to work like this. This situation is entirely avoidable, as long as you design around the incentives that influence the adjudication process.

Assessing artistic merit is awkward

I attended an open panel session for the San Francisco Arts Council (SFAC) about a year ago. Like any responsible granter, SFAC had established an evaluation process that weighs applicants according to a number of factors: artistic merit, track record, rigor of the development process or research plan, expected audience impact, reasonableness of the budget, etc. As you might expect, materials in support of artistic merit comprised the bulk of the application. I was struck, then, to see how small a role merit played in the deliberations.

In every evaluation I witnessed, artistic merit was limited to a brief opening statement by the lead assessor, essentially answering the question, “Can this person be considered a professional artist?” These pronouncements were always dry, politically correct, cautious, and uncontested, with sympathetic head nodding by the other panelists forming the extent of the debate.

Interestingly, the artists’ plans for preparatory research dominated the proceeding, lending it the air of a social sciences grant evaluation. There were lengthy discussions about steps artists could have taken to better understand the themes behind their projects—contracting non-artist experts, embarking on literature reviews, working with historical societies, bringing on advisory personnel—and the panelists used this debate as a sort of barometer for application quality.

There were many criteria to consider, but nothing attracted nearly as much discussion as the research plan. Why? Because it’s a practical way to rank applicants, and it can be done while saving face. Yes—saving face. Not the kind of self-conscious bashfulness you’d expect of teenagers at the school dance, mind you, but the kind of socialized behavior we all undertake on a daily basis in order to project our desired personae to the world, largely without noticing what we’re doing.

I’ll borrow one more example from Freakonomics to illustrate what I mean. Most penalty kicks in soccer are aimed at the corners, even though the odds of a goal are actually significantly higher if you aim dead center. So why does everyone shoot for the corners, even after you tell them what the stats say? Because kicking dead center means kicking straight at the goalkeeper. If he figures out what you’re doing, he’ll just stand there, smiling, and calmly catch the ball you shoot at him. How humiliating for the kicker! That’s why most players never even entertain the idea that they could kick to the center; everyone aims for the corners by default. Automatic face saving at its finest.

Believing in yourself is important, but this guy should be telling his players to aim for the middle more often.

Similarly, the SFAC panelists probably didn’t set out to discuss research instead of merit. But time is short, there are a lot of moving parts, the guidelines are inevitably complex, and everyone is suffering the effects of information overload. So they start to look for a way to get through the process without pulling absolutely all of their hair out, and the research plan fits the bill.

No one will judge you for pointing out methodological shortcomings in an artist’s research plan—and there are bound to be lots, given that artists are not typically known for their rigorous empirical assay design. The personal stakes are much higher, however, if you launch into an impassioned critique of some applicant’s aesthetic. You might inadvertently raise the hackles of the other jurors, only to find yourself at the receiving end of icy stares or professional hostility. Or what if the artistic practice of a particular applicant falls outside of your area of expertise? You might not want to discuss that candidate in much detail, for fear of appearing incompetent.

The research plan, on the other hand, provides a virtual cornucopia of face saving strategies. Not only is it much safer to discuss research than artistic merit, but it seems like a reasonable substitute, since all art making requires some preparatory planning. It also feels fairer to look at something more concrete: merit is highly subjective, after all, so isn’t it more responsible to keep your personal aesthetics out of the mix? Conversely, it’s useful that the research plan is not a completely cut-and-dry, non-artistic measure. After all, arts jurors wouldn’t feel nearly so altruistic if they based their funding decisions solely on the project budget. On many levels, therefore, the research plan strikes that goldilocks balance of being not too squishy, not too square, and just as penalty kicks are aimed unthinkingly at corners, it gets adopted almost automatically by the panel. Never mind that no one’s talking about art anymore.

Succumbing to false positives

But maybe it’s not so bad if the research plan plays an outsized role in grant evaluation, you say? After all, artists do need some level of understanding on the topics they deal with, so maybe it’s an acceptable stand-in for artistic excellence. Unfortunately, in the history of art, more research has rarely equaled better work. People don’t love the Ring Cycle because Wagner meticulously catalogued Norse mythology. (He didn’t. He made tons of mistakes.) They love it because he wrote some great operas that make sense even if you don’t know the first thing about Wotan or Valhalla.

For most types of art making, the research requirements range from nil to trivial. Let’s say I want to write an orchestral piece based on patterns in cosmic background radiation. Whether I spend an afternoon reading Wikipedia or a month working with an astrophysicist, at the end of it all I still need to translate those non-musical concepts into interesting sounds. There’s no guarantee that the longer research project will lead to art the resonates better with my audience, since the connections between electromagnetic noise and musical phrasing are necessarily arbitrary. (If that weren’t so, scientists would publish symphonies as well as journal articles.)

Electromagnetic noise (i.e. radio waves) from space, translated into sound. Watching this 3-minute video would be enough research for the vast majority of compositional projects.

Unless I want to precede each performance with a lecture on the subject, in-depth technical knowledge doesn’t help my art communicate. I will simply have committed the cardinal sin of third-rate modernism, placing too much focus on conceptual ideas and not enough focus on the artistic experience.

Besides, even if you completely disagree with me and do think more research is better, how would you go about policing grantee compliance? It’s probably not in your organization’s best interests to yank funding away from artists who dutifully create the work they said they would but can’t prove that they did all of their research.

What the research plan really evaluates, then, is not the merit of the proposed project, but the artistry of the grant writer. You end up funding the people who are good at producing sexy project writeups, not the people who are good at making art. Sometimes those will be the same people of course, but there’s no correlation between grant writing ability, research savvy, and talent in art making. In the best-case scenario you will miss some gems while awarding grants to a number of “false positives”: projects that look good on paper but that fall flat in the real world.

The siren song of marketing plans

This past spring, I sat on a panel with a number of Bay Area musicians, one of whom is involved with his family’s private arts foundation. Over the course of the discussions, he explained that his foundation makes its decisions based almost exclusively on the projects’ marketing plans. The board consists mostly of hard-nosed business people who feel that subjective measures like artistic merit cannot be fairly assessed. They want something tangible to base their decisions on, and a solid marketing plan seems like a responsible choice.

This is a common approach. More and more applications now ask for a marketing plan (or “audience impact statement” or similar). If one project is going to reach more people than another with the same funds, then it must be better, at least on the level of community impact. Maybe it’s not the defining art project of the decade, but hey, at least a good number of people will get to enjoy it. And if all those people are going to attend, then the art can’t be that bad.

But do those audience projections actually mean anything? The assumption is that a solid marketing strategy equals more ticket sales—which is not a very safe bet even in the most quotidian of business ventures, let alone in art. Larger presenting organizations like festivals or orchestras can usually provide realistic attendance estimates based on historical data, but most everyone else is just pulling numbers out of thin air. Marketing, after all, is not an exact science, especially when it comes to the small scale. Big-shot advertising firms all have their fair share of failures, despite astronomical budgets and the best minds money can buy. How much more so for scrappy arts organizations with audiences in the hundreds, not hundreds of thousands. So it’s worth considering to what extent these art grant marketing plans fall into the literary genre of grant writing gobbledigook.

As usual, Dilbert more or less nails it.

We also see another false correlation problem: the people who put together compelling marketing plans may or may not be able to put together compelling art. History is full of both great artists who were underappreciated in their time and talentless hacks who briefly rose to stardom before being forgotten. Perhaps you remember singer Amanda Palmer? In 2012, she became the first musician to raise over $1 million on Kickstarter (then asked her band to play for free, promptly cementing her reputation as a pariah). Having raised that kind of money, she obviously understands a thing or two about marketing, but I’ve never heard a single thing about her music.

I’m not arguing that artists shouldn’t do any marketing, but let’s be realistic here. For the vast majority of arts projects, the marketing plan is pretty much the same. Build a following through constant public showing, communicate your project among your following using their preferred channels, try to set up interviews or previews in publications that your audience follows, and put up some posters as a token gesture to the mass market. All the real work is in building an audience—and that takes years. A 150-word audience impact statement won’t make much difference one way or the other. Of course, there are certain cases where marketing innovation can play a role in the arts—Wu Tang Clan’s recent album, released on only one copy, is a good example—but most of these cases are only applicable once you already have a substantial following in place.

And while there’s not much substance to the arts grant marketing plan, the projected audience guestimate makes things even worse, because it’s a number that’s been decontextualized. No matter how uncertain the underpinnings, when you distill something down to a single number, it somehow gains unwarranted legitimacy. This is the same criticism leveled against metrics like GDP or the Corruption Perceptions Index that seem like facts but are really just one-sided interpretations of complex phenomena. The audience attendance projection will almost always be a blind guess, but it gets upgraded to the status of Official Project Estimate, with obligatory capitalization, just because it’s presented as a number. It’s a huge red herring and it pulls the attention of your evaluators away from more important factors.

Selecting for artistic merit

My purpose in this argument has not been to imply that most recipients of arts grants are not deserving. Most of them are. But the trend I’ve seen in evaluation is toward more complexity and less effectiveness—all in the name of being more objective. Sorry to break it to you, but art is perhaps the least objective of all human endeavors. If you want to support the good stuff, I’m afraid you’re going to have to address the subjective issue of artistic merit dead on. Nor is it enough to include artistic merit in a long list of evaluation criteria. By its nature, artistic merit is a sensitive topic for your jurors to broach. It will inevitably get swept under the rug if you leave evaluators to their own devices, so you have to take an active role in thrusting it into the spotlight. Some ideas:

Break the evaluation process into stages – Have jurors evaluate work samples before they look at anything else, and anonymize these samples whenever practical. You might also allow jurors to see a one-sentence description of the proposed project, to get some context on what they’re looking at, but keep non-art information to an absolute minimum in the early stages; it can be hard not to let yourself be influenced by other factors once you’re aware of them. Eliminate candidates based on artistic merit first, then reveal additional information from the applications as needed. Here’s a potential roadmap, using three rounds:

  1. Artistic Merit: anonymized work samples + short project description
  2. Mandate Fit & Applicant Capability: full project description + support materials
  3. Logistics: budget, timeline, and other non-artistic criteria

Remember the KISS principle – There will be fewer loopholes to fall into if you remove unnecessary criteria. Do you really need a research plan writeup to fairly evaluate your applicants? Does a marketing plan really give you that much useful information? I have a feeling a lot of stuff gets thrown into application guidelines in the spirit of, “Well, it can’t hurt to include it.” Yes, it can! Take a hard look at what you actually need, and revise with simplicity in mind.

Define your criteria for excellence – You’ll have an easier time getting assessors to discuss artistic merit if you define it. Last year, I served on the jury of Canada’s Jules Léger Prize (sort of a Canadian Pulitzer for chamber music). There were two other jurors on the panel, and we spent a week judging applicants anonymously over the course of five elimination rounds. When we got down to the final set, it became impossible to say one piece was “better” than the others. They were all great pieces and masterfully crafted. So we turned to the prize’s mandate: “To support groundbreaking musical creation [and] risk taking.” We tallied up groundbreakingness and riskiness across a number of artistic dimensions—form, materials, instrumentation, phrasing, etcetera—and that eventually led us to the winner without any type of professional awkwardness.

Trust your applicants – Instead of judging applicants by their marketing plans, look at their track records, then trust that they’ll figure the rest out. As Leo Lacocca said, “I hire people brighter than me and then I get out of their way.” Likewise, artists and arts organizations who have managed to get stuff done in the past will probably do so again in the future. If it’s a fantastic-sounding project and the logistical elements aren’t totally nonsensical, then just go for it.

Adopt a matchmaking role – If you receive exciting arts projects that seem a bit weak on the logistical end, consider pairing those applicants with presenters. Instead of waiting for the perfect applicant to come around, you can actively foster the collaborations that will advance your mandate. Taking this idea further, you could have two sets of calls: one for artists/creators and one of producers/presenters, each evaluated on their own terms. Then make introductions between likely pairs, stipulating the conditions under which you’d be willing to fund a joint project, and hand out your cash first-come-first-served to the pairs that respond. You probably wouldn’t want to make this your sole funding model, but it would be a valuable way to connect the dots between groups that have symbiotic needs but that aren’t yet aware of each other.

In my next post, I will discuss how to find the funding niche where you can do the most good.