When I first heard about Sonic Pixels, it was nothing more than a concept mixing audio and visual stimulants using technology and engineering. Doing a quick Google search only brought up a certain blue hedgehog and a distracting window shopping experience to upgrade my home cinema — that was it, no sketches, prototypes or examples that could be drawn from.

How would it sound? How would it look? How would it work?

And most importantly, for me, how the hell would I document this thing that hadn’t been done before?

Lewis Sykes, from Corbrook Creative, wanted to explore sound within a space. Creating a sonic experience where the listener is surrounded by the audio. Not through standard surround sound and 5.1 channel audio, but through a grid of speakers totally immersing the listener.

An exciting idea that began to develop in the lab. With the help of James Medd, the Eagle Lab Salford engineer, the idea quickly began to take shape.

Experimenting with electronics boards, such as an Arduino; an open-source hardware and software platform (Arduino, 2018). The Arduino boards are incredibly capable being used in diverse projects such as:

Using the facilities and knowledge at Eagle Lab and experimenting with different electronic components to achieve what was now becoming apparent that each device had to be powered and controlled by a computer board.

Key to the project was the micro:bit, http://microbit.org. with each device hosting the tiny computer to control the playback of audio and link to the master controller, also a micro:bit. The micro:bit is a small computing device, designed to be affordable, interactive, accessible and adaptable. For such a tiny computer, it has a built-in 5x5 LED display, motion detection (accelerometer and compass), temperature and light sensors with radio and Bluetooth wireless communication. The device also has two programmable buttons, micro USB interface and physical connection pins (for connecting devices, such as keyboards, speakers and a banana!)

All the parts began to fall into place, the speaker, the controller the input and output device, power and a way of making the pixels talk to each other. All sounds relatively simple on reflection. The interconnecting parts in creating each pixel became easier after one solution was resourced.

Another challenge became the size and budget of devices. Cheaper parts essentially led to making more pixels for the grid. Using off-the-shelf hardware can often be the quick solution to gain traction in any project, but it also comes with added cost and features that are not needed.

Rather than time-consuming 3D printing the frame for the pixel, although pretty exciting, wasn’t the solution here. Using transparent acrylic tubing with an approximate diameter as that of the size of the speaker was ideal. Cutting the tube into sections long enough to house the components while adding a rubber tube for aesthetics and safety.

This all set the parameters nicely allowing for the Micro:bit, speaker, LED lights, storage, power and custom Arduino board to be housed in each pixel. The transparent tube allows the viewer of the art installation to see the ‘working out’ of this visual, audio and sensory project. Each pixel doesn’t hide the technology that makes it work, it celebrates it, lighting it up for its audience to bear spectacle to.

As the project began to take shape, so began documenting Sonic Pixels. A video seemed the perfect choice for this, initially in a documentary to depict the manufacturing and output.

I set out to create a story documenting the development of Sonic Pixels rather than delivering facts, techno-speak and engineering terms. However, this did rely on the subject being filmed. My rationale behind this derived from observational studies suggesting the brain processes narrative storytelling more efficiently. Recollection and comprehension are more quickly realised using stories even if the content is unfamiliar to the audience. Dahlstrom describes narratives having a “privileged status” in human cognition. Storytelling allows for processing of information in the four main categories: motivation and interest, allocating cognitive resources, elaboration, and transfer into long-term memory (Dahlstrom, 2014).

Careful not to be merely preaching to the choir, as traditional documentaries have often been guilty of. Valenti explains that Sundance documentaries, such as An Inconvenient Truth (Al Gore’s lecture on global warming), Chasing Ice (an exploration of real climate change impact), and Blackfish (the effect on orcas in captivity) were successful with viewers and gained worldwide distribution. No metrics emerged that suggests social behaviour change to the topics in these documentaries. The Sundance Institute’s Tabitha Jackson describes documentaries to “pose a question powerful enough to elicit empathy in the viewer” (Valenti, 2014). Jacksons quote doesn’t elicit the audience to comprehend the subject but could lead to ‘powerful’ documentaries having an impact on the attitude of the audience towards the issue.

Perhaps a documentary was a too formal platform to depict Sonic Pixels, the project itself was more playful and experimental than the authoritative tones of documentaries like An Inconvenient Truth. I wanted to tell the story, perhaps influence and inspire the audience.

Research led towards a vlog, usually a video documenting an aspect of a person’s life. A vlog can also tell a short story, such as that of the hobbyist. It can be both instructional and entertaining usually centring around an individual (VloggingPro, 2015).

Social video platforms like YouTube and Twitch have allowed anyone to express themselves through video and talk about anything. An example of this is the Ben Heck Show; https://www.youtube.com/channel/UChturLXwYxwTOf_5krs0qvA.

A study from GlobalWebIndex (GWI) surveyed more than 17,000 internet users. From their findings, researchers found 47% of people aged 16–24 said they had watched a vlog in the previous month, this figure increased to 50% for people aged 25–34. Debunking the notion that vlog viewers are all teenagers (Digitalstrategyconsulting.com, 2015).

Vlogging is still heavily associated with entertainment, rather than reportage or documenting a complicated process. Take the Ben Heck show while
it does a fantastic job of tutorials and step-by-step guides, there’s also a significant proportion of humour in their too.

Also, take one of YouTube’s massive stars, such as Zoella, with over 12 million subscribers. Her vlogs, while informative and tutorial in nature are primarily entertainment, something you could quite easily see in Saturday Morning Kitchen — BBC One.

What happens when that trust is abused? In 2014 Zoella (Zoe Sugg), found herself inundated by angry fans after it surfaced that she used a ghostwriter for some of her vlogs and to help her write her teen novel, Girl Online (Owoseje, 2016). Fans felt bitterly betrayed that the content they loved was written and influenced by advertisers paying vloggers to promote their products.

Zoella, now clearly references that a vlog is paid for by a company. Even the title of the above video is marked as ‘AD’. This doesn’t deter her loyal fan base from deeming the content to be entirely created by the vlogger.

Taking the vlogging-style approach for Sonic Pixels and modifying this by asking questions to direct the documentation. I set up in the lab with a DSLR, monopod and a small selection of open-ended questions for Lewis. His answers drove the interview, shaping the subsequent flow during
the filming. I wanted the piece to look natural as if Lewis was talking the audience through the assembly and experimentation element of during the engineering of Sonic Pixels.

The first edit and documentation of Sonic Pixels achieved its vlogging
style. However, while engaging with engineers, developers and geeks. The film lacks in telling a diverse story to those outside of these areas. The
first endeavour failed in storytelling; unless the audience is engaged with technology, audiology or engineering, it did little to engage outside of these areas. If it were to gain any comprehension outside of these areas on social media, the narrative had to change. Reflection upon this version may be associated with the surroundings, and cultural consciousness within the context of the lab could be indicated in the documentation format. The lab is a haven for engineers, developers, 3D printing enthusiasts, technology start-ups and geeks. So inevitably this influenced the subject matter for the interviewee and the interviewer.

Charles C. Mann’s ‘1491’ eludes to preconceptions, and cultural views
are somewhat limiting. Scientific reporting on pre-Columbian America pronounced that the native Americans were child-like primitives. Claiming that the Americas were devoid of life a point of view that was unchallenged (Perrault, 2013). This representation of 15th century Americas still resurfaces time and time again with examples such as Indiana Jones and the Kingdom of the Crystal Skull and Apocalypto, where natives are treated as exotic naked or barely dressed savages. Possessing a lust for human sacrifice ripping the heart out of their victims only for the white western protagonist to put a stop to their evil-doings.

While the first edit did achieve to document the engineering behind Sonic Pixels, it did nothing to realise the final art installation the spacial sound that it created or the environment that held the art. Partly due to timing and that the latest Sonic Pixel hadn’t been actualised, sounds hadn’t been, and the festival was still some time away.

The time between these events did allow for feedback and reflection. Actually seeing and hearing Sonic Pixels would change everything. It had now become a reality I could stand watch and listen to it in all its glory. This shaped my perception hearing the booming audio disseminate from each pixel and change as you walk around. Drawing the sound on an iPad in colours all brought it together. My experience had changed, my environment changed. At the time I hadn’t realised this, but when editing, I heard my questions being replayed, my perception of Sonic Pixels had changed. I was no longer sitting in the lab, playing to the expectations of my environment and those around me. I was in Talbot Mill a 162-year-old factory producing sewing cotton and more recently mushrooms (Talbotmcr.com, 2017). The pace changed, Lewis was relaxed, to him, this was home, and to the camera, he could relay his sense of accomplishment.

Experiencing the sound, which I never expected from sub £5 speakers and a plastic tube, changed how I wanted to reflect this during the editing process. Recording a ‘wild track’ of the ambience that Talbot Mill and Sonic Pixel provided became the focus to translate to the screen.

I wanted to create a distortive experience for the view, the sense that sound changed and distorted from different angles with audio generated from NASA’s Voyager missions, giving the feeling of ethereal exploration and wonderment. To achieve this is played around with the balance of the ‘wild track’ recordings to shift the audio senses along with the visual stimulation of the visible camera pans and unfocused shots that draw the attention in, almost mystifying the audience during the first 30 seconds. Later clarity of the visual styles reveals more to the viewer yet treating shots with colour blending to highlight the Sonic Pixels playfulness with sound and lighting.

In conclusion, my depiction of science and art weaves in with other forms
of science engagement, such as museums, science documentaries and two-way communication on social media. In the hope that Sonic Pixels
can engage informally with science and non-scientific communities
through different channels, each appealing to the audience in different ways. Engineers and inventors may be inspired. Technologists could be contributors to opensource development allowing others to, more efficiently, create their own experiments. Scientists could engage with research in audiology and non-scientists may just empathise and be a little more aware of their relationship with sound and vision.

Watch the final video:

** BEST VIEWED WITH HEADPHONES FOR THE FULL AUDIO EXPERIENCE

Imagine being a listener at the centre of a powerful dawn chorus of birdsong, the creaks and groans of a haunted house or the ethereal sounds of the Solar System as recorded by NASA’s Voyager missions.

What if these sonic experiences were created not through the conventional mediums of surround sound and multi-channel audio — but rather through a matrix of remote-controlled speakers arranged in a 5x5 grid, just like pixels on a screen.

Like pixels on a screen, this system will allow you to ‘draw’, ‘paint’ and ‘animate’ with sound — to create sonic equivalents of brightness, colour and texture — through an intuitive touch-screen interface and real-time, wireless triggering of each speaker — each ‘sonic pixel’ within the grid.

Sonic Pixels has been developed by Cornbrook Creative & NOISE Orchestra, with support from Salford Eagle Labs and Arts Council England as a mini-commission for Manchester Science Festival 2017.

References

Arduino. (2018). Arduino — Introduction. [online] Available at: https://www. arduino.cc/en/Guide/Introduction [Accessed 15 Jan. 2018].

blakebevin (n.d.). Power Laces — the Auto Lacing Shoe. [online] Instructables.com. Available at: http://www.instructables.com/id/Power- Laces-the-Auto-lacing-shoe/ [Accessed 15 Jan. 2018].

Dahlstrom, M. (2014). Using narratives and storytelling to communicate science with non-expert audiences. Proceedings of the National Academy of Sciences, [online] 111(Supplement_4), pp.13614–13620. Available at: http://www.pnas.org/content/111/Supplement_4/13614.full.pdf [Accessed 16 Jan. 2018].

Digital Intelligence daily digital marketing research. [online] Available at: http://www.digitalstrategyconsulting.com/intelligence/2015/02/content_ marketing_trends_how_influential_are_youtube_vloggers.php [Accessed 16 Jan. 2018].

Kelion, L. (2016). Seven outstanding Micro Bit projects. [online] BBC News. Available at: http://www.bbc.co.uk/news/technology-35824446 [Accessed 16 Jan. 2018].

Klosowski, T. (2015). Top 10 Kickass Arduino Projects. [online] Lifehacker. com. Available at: https://lifehacker.com/top-10-kickass-arduino- projects-1747407543 [Accessed 15 Jan. 2018].

Owoseje, T. (2016). The dark side of vloggers: Belle Gibson’s cancer lie, Zoella’s ghostwriting and other YouTuber scandals. [online] International Business Times UK. Available at: http://www.ibtimes.co.uk/dark-side- vloggers-belle-gibsons-cancer-lie-zoellas-ghostwriting-other-youtuber- scandals-1558712 [Accessed 16 Jan. 2018].

Perrault, S. (2013). Communicating Popular Science From Deficit to Democracy. 1st ed. [ebook] University of California, Davis, USA: Palgrave Macmillan, pp.7–36. Available at: https://link-springer-com.salford.idm.oclc. org/content/pdf/10.1057%2F9781137017581.pdf [Accessed 9 Jan. 2018].

Sharma, A., Kadyan, N. and Besti, F. (2016). How to Make a DIY ChalKaat CNC / Laser Cutter! | DIY Hacking. [video] DIY Hacking. Available at: https://diyhacking.com/chalkaat-diy-laser-cutter/ [Accessed 15 Jan. 2018].

Talbotmcr.com. (2017). TALBOT — Talbot Mill by Capital & Centric. [online] Available at: http://talbotmcr.com [Accessed 16 Jan. 2018].

The Ben Heck Show. (2018). Mini Pinball 09: Infrared Ball Detection. [video] Available at: https://www.youtube.com/watch?v=6Jx87rpidbw [Accessed 16 Jan. 2018].

Valenti, J. (2014). Independent films communicating science at Sundance 2014. Public Understanding of Science, 23(6), pp.750–752.

VloggingPro. (2015). What is a Vlog and What Does Vlogging Mean? — VloggingPro. [online] Available at: http://vloggingpro.com/what-is-a-vlog- and-what-does-vlogging-mean/ [Accessed 16 Jan. 2018].

Zoella (2017). Gingerbread Christmas Light Cupcakes | Zoella | AD. [video] YouTube. Available at: https://www.youtube.com/watch?v=EacCdANAfyQ [Accessed 16 Jan. 2018].