\npoten<\/span>t<\/span>ial of <\/span>\u201c<\/span>dat<\/span>a-<\/span>t<\/span>o-<\/span>music\u201d<\/span> algo<\/span>rit<\/span>hms.<\/span><\/p>\n","include_cta":false,"button_text":"","button_link":"","button_style":"","give_heading_text":"","give_heading_tag":"h2","give_intro_text":"","give_fund_explorer_url":"\/give\/funds","give_fund_slug":"","give_campaign_code":"","give_default_amount":"","give_button_text":"Continue","give_read_more_link":null,"component_options_toggle":false,"component_options":{"disable_component":false,"nickname":"","identifier":"","navigable":false}},{"acf_fc_layout":"article-content","columns":[{"type":"text","text":"
<\/p>\n
By Charles E. Reineke<\/p>\n
W<\/span>hat if listening <\/strong>allowed you to \u201csee\u201d something otherwise invisible? To hear, for example, a change in the structure of a protein? A replication error in a strand of DNA?<\/p>\n And what if the sound signal you heard \u2014 this audible insight into things unseen \u2014 was wonderfully musical? A beautiful harmony indicating, say, the health of your submicroscopic life? Or, alternatively, an atonal indication that something was amiss?<\/span><\/p>\n <\/p>\n Jonathan Middleton, a composer and professor of music theory<\/strong> at EWU, has long been intrigued by the possibilities of what researchers call \u201cdata-to-music,\u201d or D2M, technologies \u2014 computer algorithms that allow users to turn data sets into musical compositions. Now, he\u2019s among the world\u2019s most influential figures in this small but growing branch of data analytics. <\/span><\/p>\n At its core, data to music is an application of what researchers call \u201csonification,\u201d the process of translating numbers into auditory images. <\/span><\/p>\n \u201cFor decades we\u2019ve been relying on visual displays of data \u2014 graphs, pie charts, tables\u2014 and that seems to be the go-to,\u201d says Middleton. \u201cIt\u2019s pretty much proven that it works, and there are a lot of fancy ways of doing visual displays now. But over the past, say, two decades, what\u2019s emerged has been something called \u2018auditory display\u2019 of data, and this allows scientists, people in business and others to actually hear their data, rather than just see it.\u201d<\/span><\/p>\n Middleton composes music that not only provides auditory visualizations, but also interpretive insights. Over the course of his own two decades of exploration, his sonification journeys have included numerous compositions drawn from unlikely data sources, perhaps most memorably the DNA of a downed Rosewood tree.<\/span><\/p>\n For many of those years, the seemingly esoteric nature of Middleton\u2019s investigations consigned his findings to relative obscurity. That has changed. These days, he\u2019s attracting the attention of a growing cadre of international scientists and entrepreneurs who, in the case of the former, see D2M as a means for analyzing and interpreting molecular-level physical phenomena and, in the latter, a vehicle for making complex industrial processes more apparent and accessible.\u00a0<\/span><\/p>\n The heart of the matter is information management. Thanks to advances in digital processing and storage, it\u2019s now easy to amass vast troves of data. Making sense of these data, however, is much less straightforward. <\/span><\/p>\n Faculty researchers at universities, along with corporate scientists in the private sector, typically deploy analytic techniques such as relational databases, machine-learning algorithms and the aforementioned graphic visualizations to interpret and act on information gleaned from their investigations. While effective, these tools aren\u2019t always enough, particularly when applied to large, complex, \u201cunstructured\u201d data caches \u2014 think molecular biology, global weather patterns, the birth and death of stars, or even the behavior of shoppers at the mall.<\/span><\/p>\n Among the first scientists to recognize the potential of D2M, or more generally, the idea of sonifying big data, was Robert Bywater, a now retired chemical biologist from the Francis Crick Institute in London. Middleton\u2019s early work on musical algorithms caught Bywater\u2019s attention as something that might provide a new way to interpret the notoriously opaque activity of amino acids, the all-important building blocks of proteins.<\/span><\/p>\n \u201cIt\u2019s like Alan Turing solving the enigma codes,\u201d he told The New York Times<\/i> in a 2016 article. \u201cYou get a message. You do not understand it. You have to convert it to something you do understand.\u201d<\/span><\/p>\n <\/p>\n \u201cIt\u2019s like Alan Turing solving the enigma codes,\u201d he told The New York Times<\/i> in a 2016 article. \u201cYou get a message. You do not understand it. You have to convert it to something you do understand.\u201d<\/span><\/p>\n <\/p><\/blockquote>\n <\/p>\n That something, for Bywater, was music. And Jonathan Middleton, with his experience in DNA-data compositions, was just the man for the job. In a paper that Bywater and Middleton later published in the journal Heliyon<\/i>, the two demonstrated that by assigning numerical values to amino acids\u2019 twists and turns within select proteins \u2014 then converting these numbers to notes\u2014 they could create a kind of \u201cprotein song\u201d that allowed listeners to easily distinguish important structures and processes.<\/p>\n More recently, in a paper published by the journal Frontiers in Big Data<\/i>, Middleton has demonstrated that similar techniques in sonification could be a game-changer for all sorts of data-intensive interpretations, including those that go well beyond the biological.<\/span><\/p>\n <\/p>\n That study was conducted<\/b><\/span> over three years with researchers from the Human-Computer Interaction Group at Finland\u2019s Tampere University. Middleton, its lead author, says he and his co-investigators were chiefly concerned with showing that a custom-built D2M algorithm could enhance engagement with additional sets of complex data points (in this instance those collected from Finnish weather records) that were usually rendered in other forms.<\/span><\/p>\n Pertti Huuskonen, one of Middleton\u2019s co-authors, is a senior research fellow with the TAUCHI Research Center at Tampere University. He and his colleagues at TAUCHI focus on \u201chuman-computer interactions,\u201d or HCI, exploring how such interactions can be optimized in real-world settings. They count a number of Finnish businesses as their clients.<\/span><\/p>\n \u201cOne goal is to find new ways to convey data to users \u2014 not just via displays and the occasional beep or buzz from the phone in your pocket,\u201d he says. \u201cHumans are pretty good at hearing sound: Almost everyone with normal hearing can distinguish loudness, pitch or direction. Trained professionals can focus on dozens of aspects of sound simultaneously \u2014 think of a symphony orchestra conductor. Because such sounds are a parallel channel to human brains, one different from visuals, this makes it worth studying in HCI.\u201d<\/span><\/p>\n Since the Finnish researchers had already pursued several sonification-related projects, Middleton reached out to them with an offer to collaborate. Huuskonen says the TAUCHI team quickly took him up on it, and soon had secured funding from the Finnish government to pursue the investigations that led to the Frontiers<\/i> findings. Middleton\u2019s background in musical composition made him a particularly attractive guy to work with.<\/p>\n \u201cWe thought that a professional composer would bring valuable wisdom on how to deliver information through sound, while still having it pleasant to listen to,\u201d Huuskonen says. \u201cAnd so it happened that Jonathan arrived to work with us for quite some time.\u201d<\/p>\n For his part, Middleton says the aesthetic sensibilities of his Finnish colleagues \u2014 their passion for making D2M sonifications \u201cpleasant to listen to\u201d\u2014 was a big part of what made the three-year gig so attractive.<\/p>\n \u201cWhen I arrived in Finland,\u201d Middleton says, \u201cmy first question to them \u2014was, \u2018Don\u2019t you just want to use any sounds?\u2019 Because that really widens the field of possibilities when hearing data. They said, \u2018Oh no, no!\u00a0 <\/span>We want to hear data with music.\u2019<\/p>\n \u201cI thought, \u2018Wow, that\u2019s amazing.\u2019 They don\u2019t just want the most efficient way of hearing their data, they want the full thing. That\u2019s when I realized that design and aesthetics were huge to them.\u201d<\/p>\n The key themes, says Huuskonen, involved \u201cfinding useful techniques for transforming data into musical structures \u2014 such as melodies, rhythms, compositions \u2014 and applying these techniques with industrial data from companies we were collaborating with.\u201d<\/p>\n The experimental piece, adds Middleton, was essentially an exercise in confirming auditory display\u2019s real-world feasibility. \u201cFirst,\u201d he says, \u201cI had to validate our work with a perceptual study. The main angle was user experience: the idea that if people heard their data with musical sounds, they might be more engaged, spend more time with it, or have deeper connections and unique perspectives.\u201d<\/p>\n Measuring user experience involved collecting survey data from 72 participants. During listening sessions at a computer, the study subjects were asked to complete tasks such as determining whether the sounds they heard represented sunshine or clouds. Other sonification exercises elicited responses to more complicated sound patterns, including melodic interludes corresponding to wind speeds.<\/p>\n <\/p>\n \u201cThe main angle was user experience: the idea that if people heard their data with musical sounds, they might be more engaged, spend more time with it, or have deeper connections and unique perspectives.\u201d<\/p><\/blockquote>\n <\/p>\n Finally, study subjects were asked to evaluate their responses to these musical data points using a variety of engagement criteria.<\/p>\n Analysis of the responses was led by a study co-author, EWU\u2019s Jeffrey Culver, a professor of business, and his Eastern students back in Cheney. The results, says Middleton, \u201cwere very promising.\u201d<\/p>\n \u201cThe paper sets a foundation for others to build sonifications of data with musical characteristics,\u201d he says. \u201cIt provides a path that begins to show which characteristics are meaningful within certain engagement factors.\u201d<\/p>\n \u201cThe International Community for Auditory Display needs a paper like this to move forward,\u201d Middleton adds, referencing the organization that promotes research in sonification and related areas. \u201cThere has been a gap between those who think sonification can include musical traits and those who think music is problematic,\u201d he says.\u00a0<\/span><\/p>\n Middleton, as a composer and artist, is insistent that music is not at all problematic; that instead, in both form and function, it is a perfect means of making auditory display an even more powerful scientific tool. <\/span><\/p>\n \u201cWe humans,\u201d he says, \u201csee improved functionality in things that are attractive.\u201d<\/span><\/p>\n <\/p>\n \u00a0<\/span><\/p>\n","pull_quote":"","image":false,"caption":""}],"component_options_toggle":false,"component_options":{"disable_component":false,"nickname":"","identifier":"","navigable":false}}],"page_override_title":""},"_links":{"self":[{"href":"https:\/\/www.ewu.edu\/magazine\/wp-json\/wp\/v2\/stories\/2509","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.ewu.edu\/magazine\/wp-json\/wp\/v2\/stories"}],"about":[{"href":"https:\/\/www.ewu.edu\/magazine\/wp-json\/wp\/v2\/types\/stories"}],"author":[{"embeddable":true,"href":"https:\/\/www.ewu.edu\/magazine\/wp-json\/wp\/v2\/users\/484"}],"version-history":[{"count":19,"href":"https:\/\/www.ewu.edu\/magazine\/wp-json\/wp\/v2\/stories\/2509\/revisions"}],"predecessor-version":[{"id":86119,"href":"https:\/\/www.ewu.edu\/magazine\/wp-json\/wp\/v2\/stories\/2509\/revisions\/86119"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.ewu.edu\/magazine\/wp-json\/wp\/v2\/media\/2512"}],"wp:attachment":[{"href":"https:\/\/www.ewu.edu\/magazine\/wp-json\/wp\/v2\/media?parent=2509"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}![\"Kenia](\"https:\/\/www.ewu.edu\/magazine\/wp-content\/uploads\/sites\/12\/2024\/01\/2023-11-27-John-Middleton-Shoot-29-copy-200x300.jpg\")
\nThe work that became<\/b><\/span> the Frontiers<\/i> paper was undertaken with the participation of five Finnish corporations, each with an interest in potential D2M solutions.<\/p>\n