STRATEGY
A look at 58 years of Billboard Hot 100 uncovers the secret ingredients of popular music.
Popular music has the reputation of being canned, overly engineered ear candy; the electro-pop duo The Chainsmokers may come to mind. Some say that the current popular music landscape has nothing but manufactured pop and hip-hop hits that all basically sound the same. But it could equally be argued that every generation thinks the following generation’s music all sounds the same (and is not particularly good). A historical look at Billboard Hot 100 charts from their beginning in 1958 to 2016 actually supports the belief that the mainstream music we listen to in a given year all tends to be reasonably similar. But the breakouts—the songs that hit the very top of the charts—are those that are a bit differentiated. One recent example comes from oft-derided Justin Bieber, who did not have a #1 hit until his 2015 track “What Do You Mean?” Since this song, however, he has claimed the top spot twice more on his own, to say nothing of his involvement in recent smash “Despacito.” What’s different? His most recent album—the one that generated multiple #1 hits—had songs that were more atypical, compared with the current crop of popular tunes, than his previous tracks. Conversely, Michael Jackson had a slew of Top 5 hits off his record-breaking album “Thriller”. However, the 6th single released from the album “P.Y.T. (Pretty Young Thing)” reached #10 on the Hot 100, and spent just four of its 16 weeks in the Top 20. It was the most typical of the singles from the album, sounding most like the other songs charting at the time. Defining popular What makes popular culture popular? Academics, critics and just about anyone dialed in to pop culture has an opinion. We know, for example, that artists’ characteristics and marketing budgets are important for driving success in creative industries, but also in the mix are the audience, the external environment and the competition. Historically, academics looked at popularity from the position that social structure determined cultural output, as well as its success or failure. While few would deny that social context matters for the success of cultural products, the content of those products—movies, art, music—does as well, yet has not been explored as rigorously as it should be. Fortunately, big data, machine learning and other methodologies from computer science are now being brought into the social sciences. This means that we can examine what makes pop culture popular from a more content-focused point of view. Doing just that, Michael Mauskapf and I find that counter to the idea that all hit music sounds the same, songs that sound more like what is popular (or “typical”) in a given year have not performed as well throughout chart history as those that are a bit different. In our article, “What Makes Popular Culture Popular? Product Features and Optimal Differentiation in Music”, forthcoming in the American Sociological Review, we find that the songs that reach the highest echelons of the charts bear some similarity to other popular songs that are out at the same time, but they must be unique in certain ways. That is, they must be optimally differentiated. Acoustic footprints To analyse the similarities in pop music, we collected data from The Echo Nest (now owned by Spotify) on the nearly 27,000 songs and over 6,000 artists that appeared on the Billboard Hot 100 charts between August 1958 and March 2016. Songs are unique creations, but they have certain features that can be defined and captured: Each song has its own “acoustic footprint” which can be used to compare it to other songs. The Echo Nest captures such features, and in our analysis we used each song’s key, mode, tempo, time signature, acousticness, danceability, energy, instrumentalness, liveness, speechiness and (emotional) valence. Some of those will be familiar to anyone with a passing knowledge of music; others are the company’s creations for classification and comparison purposes. Songs are bundles or combinations of these features, so rather than comparing songs’ individual attributes, we devised a “typicality” score that considers each song as a holistic, single entity. The acoustic footprint of each song is compared to that of all the songs that appeared on the charts in the year prior to its release. We use a cosine similarity measure to capture the distance between each song pair and then average those distances across all the songs from the previous year. The resulting average provides the most systematic attempt at determining how much a given song sounds like the rest of the charts at the time of its release. To our surprise, we find that songs with a below-average typicality—i.e. those that are “optimally differentiated”—tend to do better on the Hot 100. Optimally differentiated songs have some combination of acoustic features that is sufficiently dissimilar, but not too different, to that of the other songs released in the previous year. Perhaps the tempo or energy is unlike other songs on the chart around the same time, while other features are relatively similar. Such a song would be optimally differentiated: different enough to stand out, but not so much so as to alienate large groups of potential listeners.
The preceding chart compares the typicality of Top 40 songs to all songs that did not reach the Top 40 on the Hot 100, and demonstrates that there is not a profound difference between these two groups. Yet in most years, Top 40 songs are slightly more atypical on average than the rest of the charts. On the other hand, the following chart shows how #1 songs compare to all other charting songs in a given year. Though more variable, songs that hit #1 are generally less typical, often substantially so.
We think optimal differentiation throws a wrench in the works of the idea of “hit song science”, which suggests that top songs can be reverse-engineered based on what audiences are more likely to listen to or buy. This idea neglects the fact that the other music out at the same time is an important factor in a song’s or artist’s success. Writing a song specifically to hit the top of the charts requires some idea—at best an educated guess—of the type of music that will be out and popular by the time it is released.
What becomes popular next is likely to be slightly differentiated from the last round of hits, leading to a constant evolution of what is popular. Popularity is a moving target, but the context always remains relevant.
Meaningfully memorable songs
In performing our analyses, we accounted for variables such as an artist's previous success, whether a song was released by a major or independent label, and other elements that could account for a song’s chart performance, like its genre. Rather than relying solely on the acoustic footprint, we worked to situate content within context to more clearly demonstrate the specific value of songs’ acoustic content.
Ultimately, while it may be possible to reverse-engineer a song to generate a high degree of popularity, doing so neglects the competitive context of the time. Artists can’t know what songs other musicians will release at the same time, nor exactly what audiences will find to be optimally different. For those striving for a #1 hit, it pays to be a little unique—but not too far out there: Songs still need a hook for listeners to grab onto. As with most innovation, this is what makes popular culture popular: the right balance between familiarity and novelty. Finding the appropriate level of differentiation in a given competitive context is at least as much art as it is science.
Noah Askin is an Assistant Professor of Organisational Behaviour at INSEAD. You can also watch a TEDxINSEAD talk about an earlier version of this paper.
This article is republished courtesy of INSEAD Knowledge. Copyright INSEAD 2017.