This week, tie on those sneakers and jog along with Tristra Newyear Yeager and Mansimran Singh, Lead Engineer at Weav, a company that applies adaptive music and audio to fitness.
Mani digs into the challenges Weav faced in getting the music business on board with adaptive music for their original running app, Weav Run, one of the first to attempt to make music match listener's activities in real time. How can new tech change the way we train? How can the music business race to capture this important market?
Find out what startups need to understand about music licensing when they are looking to incorporate music or stems into their products. How did Mani’s experiences with Weav Run inform Weav’s new project Magic Run, a virtual running coach app that uses AI to generate personalized running workouts and provide real-time feedback to runners of all abilities and goals?
Looking for Rock Paper Scanner, the newsletter of music tech news curated by the Rock Paper Scissors PR team? Subscribe here to get it in your inbox every Friday!
Early Bird Tickets are on sale at a special price until August 15:
Listen to the full episode here on this page, or wherever you pod your favorite casts.
Listen wherever you pod your casts:
Find a full transcript of Tristra and Mani's Conversation below:
Tristra: Hey everyone. Welcome back to Music Tectonics, the podcast that goes beneath the surface of music and tech. I'm your host, Tristra Newyear Yager, chief strategy officer at Rock Paper Scissors, the music tech PR firm. For this episode, we're continuing our very, very informal exploration of the intersection of music, tech, and wellness, and talking to Mani Singh of Weave Music.
Mani is lead engineer there, using his extensive background in building apps and other cool things. Most recently, an app that knits AI and better running training together called Magic Run. And we'll talk about that in a second. Though Magic Run isn't connected directly to music at the moment, it's predecessor we've run featured adaptive music for runners, and we'll get to talk to Mani a bit about his experience building that product. So hi, Mani. Thank you so much for joining us today.
Mani: Hey, thanks for having me. It's great to be here.
What is Magic Run?
[00:00:51] Tristra: So first off, what do you usually say if you're explaining what you do to people outside of tech? Like if you run into someone at a coffee shop or, at a dinner party, how do you explain what you do?
Mani: It depends on the current product I'm building, but at the moment I tell people I make a running app to make running easier. I really want to take the guesswork out of your running. So when you open your phone and you're like.
All motivated to go out what you're going to do. You kind of have an answer and I think a lot of the times people feel lost on kind of the work that they should execute and we're trying to take as many factors into account as possible, based on what you did last week, you know, how many steps you've taken, what time of the day it is and all these things and kind of eventually craft you the perfect running workout for now.
Tristra: That's really awesome. Now I've had a chance to use Magic Run a little bit, but I didn't realize there were all those inputs going into it. So you know, is it cool, can we talk about Weav Run a little bit? I know there had that was a really really neat product with a beautiful concept behind it one. I really appreciated personally But I know there was some moments there and you've pivoted away from it. Can you give us a quick and simple overview of what the role that music played in Weave Run?
[00:02:06] Mani: So at Weav, we've been creating adaptive music technology for about seven and a half, eight years now.
We started off by creating an adaptive music engine that could take stems, which are like the building blocks of recorded music and publish them using our production tool, and you could attach alternative stems to it. So whenever a different input signal came in, let's say, like, your steps per minute, or the change in energy or intensity, we could, at the most basic level, switch out the stems that we're playing, essentially creating a remix of the song. And we did this all in real time. So in Weav Run, which was our first product that we launched which is a full featured running app in the U. S. app store. There's about 600 songs in there from record labels like Sony, Warner, and a handful of Indies where our music engine inside the app will remix the song based on your running cadence. So every time your foot hits the ground, the beat will match your feet. And we started playing with intensity of the song, the intensity of the remix. And music just kind of lends itself to this rhythmic activities, like fitness, especially things like cycling and running.
So that's where we started.
Explaining the Magic Run Model
Tristra: So when you were thinking about adaptive music back in the day, when you're first starting to work on this engine, what were, how did you go about explaining that model to people in the music business? That must have been a complicated conversation.
[00:03:33] Mani: So that was really, really hard.
And it still is because adaptive music is so flexible. It's such a broad term. But two challenges we had was how do you explain adaptive music? And then how do you explain to people that, hey, we actually don't want them to produce records anymore. We want them to produce recipes. And I explain that in a second.
So, adaptive music has been around for a long time. Video games have had adaptive music since basically video games have had sound where, you know, things you were doing in a video game will change what would happen. And we wanted to bring that to real life human experiences. And that's where we kind of started.
And our approach to doing that was to not use generative AI, but instead build the tools that allows producers to write the recipe for how a song should be performed when it's being listened to. And so that recipe with a bunch of alternative stems goes into this music engine that works across all platforms and a developer could then take that and attach it to your running or attach it to your cycling or attach it to, you know, a fitness activity or a video game.
And so we experimented a lot in digital fitness using that. We did some projects with Peloton and Red Bull which was kind of exciting. Peloton became one of our investors eventually. But the challenge that we kept running into was trying to convince the recorded music industry that they might not have control over the final master that gets played to the user, which was a completely new concept.
Because, once you bounce out a final master, you kind of know exactly how it's going to sound every single time. And we were saying, well, hold on a second. How about we put together a recipe so that it's a performance for the user in real time.
Advice for the Beginning of the Startup Journey
[00:05:20] Tristra: That is so interesting. I think you've stumbled into one of the lesser discussed, but more important aspects of recorded music, copyright, whatever.
I mean, people focus on the commercial side where they're like, Oh, you know, there are people are violating copyright. The artists aren't getting paid. Of course, that matters to people, but what matters a lot to artists and producers is control, right? And in some ways, you're asking them to see their control slightly differently or think about music in a way where they don't have immediate or sort of like the final locked control, although they can obviously control the inputs and could play with the recipe.
Right? So that's a really delicate moment that you were, or sort of a, how should I say it? You're trying to thread the eye of the needle in a way that I could imagine would lead to some long convincing, right. So some difficult meetings. But you know, you guys did pull it off and you do have a catalog of songs or of tracks that people can adapt to their activity.
Based on your experience, what advice would you give startups today or engineers who are looking to make something that involves recorded music, that is a new use case? What do you wish you'd known as you started on this journey?
[00:06:34] Mani: I think we went through so many you know, firsts in the industry. We were 1 of the first to kind of essentially push labels to treat stems as a product in itself.
And they hadn't really considered it. We were the first to kind of integrate, I think, with dedex feeds to receive stems instead of masters and things like that. It's a little bit different now, and I think we've seen the shift in the industry around that being more flexible to alternative revenue streams.
And it might be partly because streaming is getting squeezed and you know, revenue per stream is on the decline. But anybody who's coming into this sector know that it is always going to be a challenge and you might not get the songs that you really want for your product. But we are seeing a push from Warner who've been really, kind of championing and building out services. And I think Warner is partner with a company that I can't remember the name of where you can just go and kind of license a huge catalog of their music.
Tristra: Yeah. It's, I think it's part of feed.fm and they allow people to play around. I mean, it's more than like just a little sandbox where you can only have like two tracks to play with, but it's a limited catalog, but it's big enough to really have some fun with. But yeah, they have created a space for developers, which is really cool. And we'll put that in the show notes, everyone.
Mani: The challenge actually with feed.fm and so there's licensing is a huge issue or, or a challenge, let's say, and you don't necessarily get the kind of licenses you want as a developer of a new product. I don't know how much people in your audience know about like music licensing, but
Tristra: Probably more than almost any other podcast audience. So you can, you can feel free to get into the weeds if you'd like.
Mani: Yeah. So one of the reasons why we brought up as US only is because although we were able to clear masters worldwide for majority of our catalog, we kept running into publishing issues you know, across all these jurisdictions.
And that's just the thing that's really, really hard to scale. So if you're going to build a global product, you need to make sure you have a catalog that you're happy with, that you can clear globally. And I think it's going to get easier and there are going to be automated services to help you to kind of achieve that.
But that was, that was a big pain point for us for basically our hands were tied. We couldn't launch our product anywhere else. That's 1. The 2nd is radio licensing versus what we were trying to do. So a lot of what the industry had seen at the time was, you know, radio licensing, anybody who had a service would just get a feed on FM kind of integration, and you just get a bunch of songs, but we wanted our users to be able to pick the music that they listen to in the order that they listen to which had technical issues. So, our songs are really good at remixing into each other.
So, if you were out running, they seem to go from one to the next. So we put a lot of effort into making that work, but also, it kind of shifted how we had to think about licensing. And that was another challenge that we had to get over. So I think it'd be useful to have somebody who understands licensing so you're not going into this completely blind and we spent a lot of time and resources learning from the inside out.
What Inspired Magic Run?
[00:09:47] Tristra: Yeah. And there are lessons you're like, I don't know why I have to learn this, but I've got to learn it. And this is, it can be a very difficult learning process. That's amazing though.
Thank you for being so forthcoming with your experience. I think that's great advice. So we're going to talk a little bit more about magic run in just a second, but first we're going to have a little break.
We're back. So, Mani, thank you so much again for sharing your experience working with the music industry. At the moment, you're not having to do that particular work. You've pivoted away with Magic Run. Tell us a bit about how you decided to create a virtual running coach as your next project because it's pretty intriguing.
[00:10:24] Mani: Yeah. So while we were working on Weav Run, sort of the previous iteration of this running app we invented adaptive running workouts. So we're all about adaptive audio experiences. Cause I'm a firm believer of augmented reality being an auditory experience instead of a visual one, because it's less restrictive.
So it can kind of slide into your life way easier and better. So we invented adaptive workouts where we took human voice and we were able to stitch together in real time to give you real time feedback. And so it was essentially a podcast that was like generated for you kind of thing while you were out running. And we put out, I think a dozen or so of these workouts that we've run. But while we were trying to think of, you know, what happens with our products that were out there, we sort of stumbled upon this idea that, okay, what if we took the adaptive workouts that we had sort of invented for this before we run and scale them. Where we would own all the content and have similar sort of goals that we did for our music where we want to deliver personalized experiences, but this time we don't have the content licensing restrictions or, you know, market restrictions or anything like that. So we have a voice, we work with Kelly Roberts who's a running coach and running celebrity, I would say. You can find her on our social. So we use her voice and we pair that with her personality. And so we are training AI models to essentially generate a script, like a podcast, for you based on your running goals and the running plan you're going to execute and then have that be read out in her voice. And it will read out your real time stats and, you know, tell you that maybe you should wear sunscreen today before heading out because the UV light, but it's really just you want a friendly coach and I think with the kind of state of the art of AI, we're finally at a point where we can do that at scale for every individual runner who comes in.
Tristra: I found it really interesting how you struck a balance. Having used Magic Run a little bit, you struck a good balance between, you know, having the coach quote unquote, present in your ears and just giving the person time to like just run, right. Especially if you're doing sprints, the last thing you want to hear is somebody going, come on, go, go, go, go.
You're just like, ah, just let me suffer through the next 90 seconds or whatever. So, how did you find that right balance? Like, was it just experimentation or, how'd you go about homing in on that?
Mani: Yeah. So I'm a huge champion of using your own products. So while we were building, we've run our whole team got deeply into running.
I think between us, we ran like 20 half marathons and a handful of full marathons. And so we went through the, you know, 16, 20 week training programs to really understand what it is that people want or what it is that we want. And when do we feel the most lost? And when is the right time to have the right information?
What kind of feedback do we want? And it's a continuous process. So I run, well, I started running again after sort of a two year COVID break. And so, I was the first user of magic run and everything that we put on the Magic Run is either directly requested by our testers or is something that I really, really want.
So I think last week we launched our Apple watch app, which means now you can put away your phone. And so you get all the feedback of how much time you have left both in your ears, but also on your wrist. So, you know, what's upcoming and things like that. Next week's update we're going to launch open warmups, which is something that people have been requesting for a long time.
So our AI generates currently the right length of the warm up based on the workout that it creates for you. But a lot of the people like to do some sort of a warm up and then do stretches and whatnot. And so now you'll be able to kind of do that and then go do your intervals and then we really want to tap into as you're probably aware of, if you're a user of Magic Run that the voice input and audio feedback. So in Magic Run, you can request a workout by just speaking to it. And eventually you'll be able to record how you felt during the run without having to write down your notes because I do some of my best thinking while, you know, after having run like 5 miles or something.
I'd like all these ideas pouring in and I have nowhere to record them. And now you can kind of record all of that in Magic Run and categorize it and see how you progress.
What Workouts Are Users Doing?
[00:15:03] Tristra: That's really cool. And I can imagine, I mean, this is kind of messed up for work life balance, but for someone who's also a massive users, things like Slack, and I often Slack myself really weird ideas just to keep track of them, I could see you could have a little integration here and there with other places to keep notes. So to capture those moments where you're having that amazing feeling post run and you're like, oh, yeah, we need to have a cat symphony, that's what we need to do or whatever. Anyone who wants to have a cat symphony can use that idea, by the way.
All right. This is really interesting. Now I realized Mani, you and I are talking as people who enjoy running and working out. Maybe we want to say just for a second, why, you know, for non runners, this may seem a little baffling, like, don't you just like go and run. But there's actually many different ways you can progress as a runner, whether you're focusing on endurance or speed.
And you can do fun things like sprint for the 90 seconds and then have a recovery period. So in some ways, keeping all this in your head while you're trying to run really fast is really difficult. So, do you want to just give us a little bit of an overview of the kinds of workouts that you've seen users enjoy so far?
Mani: Yeah, so running can be really, it can take a lot of motivation to get up and go for a run. And it can be really hard. I totally understand that. I think if you're a beginner runner you know, if you're just getting out there during your walk /run kind of cycles, you're already a runner, right? Like, that's how we think about it. You don't have to do like a half marathon every week to consider consider yourself a runner. And running is the easiest sport to get into. And it has a long way to go and a really supporting community. Like, I've never seen a sport where 50, 000 people show up and everybody gets a medal.
Like, that's just never happens. And so we've seen people execute the usual kind of like tempo runs and pace training and things like that within our within our app, but also we can generate fun intervals that other training programs can't do. I don't know how beneficial they are if you're training for like a particular race and you have an instructor telling you to do specific things, but because we can mix and match intervals that are like, you know 90 seconds or 400 meters or whatever. And it can be a lot of fun. And with our audio coach, it's always in your ear. So you can listen to your own podcast or music, but the audio coach will come in and tell you you're almost there.
And actually we increased the amount of motivation you get as you get closer to the end of your workout. Cause that's where we know that you might need that extra little push.
How the AI Behind Magic Run works
[00:17:42] Tristra: Amazing. Thanks. I think that'll help people who have yet to try running or who may never want to try running, understand sort of the basics of why magic run is really interesting.
So can you give us a lay person's understanding of how the AI behind Magic Run works? How would you explain it to an educated, but somewhat lost lay person?
[00:18:07] Mani: So we're trying to think of it as three separate goals that we need to accomplish and get good at. One is what is the workout that you should do today? And you can go hire a running coach and that they can solve that problem for you.
But I think we can take data from your health, Apple health or whatever, and really see what, what your stats have been over the last, let's say, 90 days and give you the right thing at the right time. So, it takes the guesswork out of what is the thing I should do today. The 2nd thing we're doing is making sure that we say to you the right things at the right time which might be that you're, you know, got 30 seconds left in this interval or count you in and just count you out. And tell you little things like the sunscreen thing. I really need that. Like, I need that all the time. I step out and I'm like, it's really hot and I need sunscreen. I live in Europe, so.
Tristra: It's a huge problem here. And I keep laughing because it's the very thing that I always forget.
Mani: Yeah. And so you don't want to feel that working out is like a tedious activity. So you have this, like, almost an assistant that will tell you, okay, here's like your checklist that you, that you've got to get done.
Or you know, maybe today if you're running a five mile, you can try a new course. When things like that, so I think all these things that we're going to introduce and the 3rd thing is turning that text into voice. I think we're seeing a lot of voice based products coming out and a lot of like, chat based products coming out.
And we're finally at a point where computer generated voice sound like human generated voice and we can train custom models. So we're experimenting with using human trainer voices and training a model so that when that full package of these 3 goals is delivered to you, you feel like you have that personal coach with you all the time.
The Biggest Challenges in Building Compelling Apps
[00:20:01] Tristra: It's definitely more motivating than a glitchy, strange robotic voice. So that could be fun too. All right. We're going to take another quick break and then we're going to talk more generally about apps and fitness.
Okay. We're back here with Mani Singh of Weav Music and Magic Run. And so Mani, what are some of the biggest challenges to your mind in building compelling apps for fitness?
Speaking more generally, we talked to a lot of detail about running in particular, but what are some things you think people need to keep in mind as they approach the problem of how do I build a good app for fitness purposes?
Mani: I think fitness is an interesting space to be in, because you're usually trying to motivate people to do a thing that's good for them, but there's just super reluctant to do it.
Like, I'm not motivated to get up and go for a run all the time. But then I look at it and I go, like I've been sitting all day, maybe I should, like, take a walk or whatever. And habit building and fitness is basically the most important thing. Over, like, the last 50 years the innovation and fitness has come from, besides what, you know, the academia has kind of learned about how it should be delivered the sort of the consumer products and fitness have always been about making it more fun and I think for an amateur, let's say runner, it should be about making it more fun and just providing you with all the help you need to get you to your goal. And so that's why I think personalized fitness, and we're going to see a new era of personalized fitness happen over the next decade, which is going to be super exciting.
And I think the platforms that have lots of content right now, who put a lot of effort into producing content are just not able to scale it and personalize it to the level. That apps like ours might be able to and I think that's going to be a challenge for them to solve on how do they get.
individual, how do they target users and help them achieve their individual goals? And I think there's lots of opportunity to build personalized fitness experiences now.
The Future of AI Personalization
Tristra: Yeah, that's such a cool point. And you know, what always intrigued me about Weav Run from the beginning was this sort of adaptive and personalized version of music that it created.
Let's think a little bit more generally, though. What do you think the future of AI personalization looks like? How will apps like yours that help people customize their workouts and goals and just figure out what they want to do on their next run evolve in the foreseeable future? How sort of where are we going with all this?
[00:22:31] Mani: I think, oh, this is an interesting question.
Tristra: I try.
Mani: I think things start to really work when they integrate with each other. So your work on Apple tell you what to do based on maybe what you ate in the last week, like you go on a vacation for a week and everything goes out of the door.
Tristra: Your daily ice cream habit means you definitely need to go for a half hour run today, my friend. Right?
[00:23:02] Mani: Yeah. But one of the other things that we're experimenting with and that I'm personally super interested in is how do we balance scheduling of fitness? So we're experimenting with looking at your calendar and finding you the right slot and then telling you, Hey, maybe you got an hour there so you can squeeze in a half hour run because it'll take you, it will take you like 15 minutes to shower afterwards.
And so we really need to make sure that all these tools work together and are able to like, help make your life overall better. And I think so all these siloed apps and all these platforms that have, you know, they do one thing really well are going to have to start partnering with other platforms to help make sure that the person gets the best experience.
Tristra: That idea of scheduling is really intriguing and you know, it's been a long time piece of advice I've seen floating around, just like, make sure to get it in or schedule it, put it in your calendar. But if it, if your app did that for you, that would be pretty slick.
Mani: Yeah. And it's something that is technically already possible and it's on our roadmap.
So you'll eventually see it in Magic Run. It's on our roadmap along with training plans. So as we start to roll out sort of eight to 16 week training plans we'll also work in scheduling.
What Kind of Tech is Exciting Right Now?
[00:24:21] Tristra: Amazing. So are there other apps or ideas or products out there that you find really intriguing could be in fitness or wellness, or could be totally beyond that?
What kind of, what kind of tech are you looking at and finding exciting?
If you have time to think about any other app at your own.
[00:24:41] Mani: So I think staying in the world of fitness I'm quite excited about how Peloton and Apple and actually a small company called Lupa are doing digital fitness. Lupa's focus has been on running and mindfulness during running and really stop worrying about performance.
Cause you know, not everybody needs to go run. for a particular time. And we're seeing even training plans. So with the trainers that we work with, we're seeing training plans being developed around the idea of running for time instead of running at a particular pace, because most people don't want to race.
Most people don't want to compete. They just want to make sure that they are healthy and they're able to have fitness work for them instead of them having to work for fitness. And so I think this holistic approach of taking into account and your putting you on kind of on the right path and making sure that, you know, you get your daily activity in is probably the right approach.
Tristra: Yeah, I think Apple's new fitness app with this pesky rings is remarkably motivating. It's so simple. And yet you've really like, even someone like me, who's an extreme, I mean, I'm a Gen X or so I'm extremely eye roll about everything. But it, it does get me to do like, all right, I will do one more five minute brisk walk for you app, you know?
So I can close my little ring. It's very interesting how those small, but like really sticky things can help you just get a little bit more in and I mean, and in a pretty enjoyable way.
Mani: And I think a lot of the fitness apps you know, of the, let's say the previous generation were about tracking.
They were about tracking your statistics and your numbers. So you already had to like come in with the mindset that I'm going to do something and I just need some help instead of like, you know, writing a log in a book my phone will do it. But a lot of people who want to get into fitness feel lost where to start.
And a lot of so that on ramp off, you know, getting into an activity is what we hope to solve with Magic Run. So, when you first come to the app, we can just get you started without you having to worry about where this would go, how this will work out later down the line and kind of handhold you and work on habits forming and things like that to make sure you achieve your long term fitness goals.
[00:27:06] Tristra: Yeah, it's, it's kind of interesting in some ways, there's a real parallel between music and fitness technology in that the sort of first generation of digital music creation products were basically copy paste of a studio you know, a studio set up, right. So like a board in a studio and on the fitness side, it was like what coaches would use.
So there was the sort of the stuff that had trickled down from the professional coaching world, but most of us aren't elite athletes. And so what we might need is something very different. And so the, the form factors and everything needs to be just a little different and doesn't have to be quite so like serious and all, and, you know, metrics don't really matter to most of us when it comes to beyond some really basic ones when it comes to fitness.
So it's really cool to see how things are evolving and becoming kind of. Way more user friendly and kind of no longer focused on traditional sports slash coaching norms. It's really cool.
[00:28:05] Mani: Yeah. And I think another overlap in the music and some of the digital fitness sector, at least, is recommendations.
And just answering the question, you know, what should I listen to today? Or what should my workout read today? And it becomes a huge and intimidating challenge to try and determine yourself. Like, what is it that I should work on today? And you don't need to go, hire a coach or whatever, like you, all the content exists already.
There's so much content out there and it's like, how do you filter through it all to find the right thing for you at the right time? And it's an unsolved issue in music. It's an unsolved issue, you know, in in video, like I said on Netflix and I'm like, I want to watch something and I don't know what should I watch today and I've already wasted half an hour.
Right. And I think whoever is able to take that guesswork out of you, it's just going to make the experience a lot more enjoyable and hopefully we can achieve that in Magic Run. And I think, if you open Magic Run now. ,It already sort of goes like, okay, what's on today, right? What is on your mind?
And you can go and ask, Hey, I have 30 minutes. Give me a 30 minute workout. Or if you have a particular set of intervals you want to run, you can say, I want 400 meters with 90 seconds rest, run 8 times. And it will just create that for you. And eventually we'll be able to suggest things on the workouts you should run today.
And also tell you like to take a rest day. But also what to do on a rest day, because if you're like me and if you take a rest day and that turns into like a rest month, you want to make sure that you stay in a steady habit of doing something.
Tristra: That is so cool. Well, thank you so much Mani for your time and your insights. I really appreciate it.
Mani: Thank you for having me. This was so much fun.
---
Let us know what you think! Tweet @MusicTectonics, find us on LinkedIn, Facebook and Instagram, or connect with podcast host Dmitri Vietze on LinkedIn, Twitter, and Facebook.
The Music Tectonics podcast goes beneath the surface of the music industry to explore how technology is changing the way business gets done. Weekly episodes include interviews with music tech movers & shakers, deep dives into seismic shifts, and more.