The New Interactive TV Experience

Geof: Welcome to our latest arm viewpoints podcast. Our topic for today, deals with having fun and specifically the kind of fun that you can have when you’re playing great video games on your TV. Gaming on TV has come a long way from the days of pong and space invaders played on crude consoles hooked up to your TV.

Today’s smart TVs are online by definition and contain enough great technology to deliver amazing gaming experiences with or without a console system being attached to them. It’s part of a broader evolution of TV. As we moved from over the air channels to on demand to streaming content delivered digitally, and technology has even changed more traditional TV experiences, such as watching sports.

You can. For example, watch a game in high definition 4k and enjoy the option of multiple camera angles. Getting back to gaming, new digital TV sets now include third party gaming service. Built in. And in some cases they provide smart cameras for video conferencing and other uses interactivity is an integral part of this TV experience from choosing what to watch next, to choosing what happens next in choose your own adventure shows.

In this context, the convergence between the TV content world and the gaming world appears to be inevit. I’m very fortunate to be joined today by three guests who are highly qualified to explore this convergence and what it will mean for anyone who wants to have fun with their TV. Our first guest is Pablo frail director of the home market segment inside the client line of business at arm he’s responsible for business growth opportunities in the TV streaming smart display and game console markets.

Also here today is Ralph Hauwert, senior vice president and GM for core unity and cloud at gaming innovator Unity. Ralph previously held the title, the senior vice president of R& D, where he ran Unity’s world class core engineering and product teams globally in building and delivering the unity editor and engine to all creators.

Rounding out our panel is Tetsuya Kimora, distinguished engineer system architecture design at Sony. He started in the software development business on the design of user interfaces in product development of camcorder and digital still cameras. He’s also been engaged in overall architecture design throughout the framework and operating system areas as the progress of software and system development expands with the digitization of recording and playback media.

Welcome to you all. Can you tell us a bit more about this market and the opportunity it’s presenting and I’ll kick us off with asking Pablo for his thoughts. Yeah.

Pablo: Thanks, jeff. I think the TV and gaming market presents a really interesting opportunity. This says the reality is there is about 3 billion gamers in the world now including all mobile gaming and console gaming and PC gaming.

Traditionally people game playing on consoles or playing on the TV, had to have a console attached to it. But now everyone who plays games on mobile potentially can become a gamer also on a TV. So that presents a huge amount of opportunity for the industry. The other thing to think about is the size of the TVs these days being sold.

I think the average size these days is close to 50 inch.. And you’re selling about 200 million units or more every year. So you can imagine the amount of screen and screen time that, that we can, that is gonna be enjoyed in houses worldwide that present a huge amount of opportunity for the industry, for the gaming industry and for the TV industry in general.

Geof: So what’s your perspective on it, Ralph?

Ralph: Well, I think TVs are, if you kinda look at the lifetime of TV at this point in time from inception to where we are today, traditionally, actually a very slow evolving consumer facing screen. Right. Like if you look at the revolution of what happened in smartphone, starting somewhere into the mid two thousands to where we are today, and you look at the same evolution and span of evolution of TVs, it’s a really slow evolving medium in which most of the things we’ve seen have been about qualitative improvements around screen.

We added color, it became 4k people try to add 3d, but the most significant change that has happened is. What content can be consumed on a TV. So you mentioned in your opening, attaching a game, console to your TV at that point it’s a monitor. It’s not a TV. And if you kinda look at what, what has changed over even the last 10 years, or maybe even the last seven years, the channel up and down button, the most pivotal thing to interact with your TV is no longer being used in the same way, because the TV is now portal to content streams, but those content streams are actually still fairly, traditional the traditional nature of TV is a channel. A stream comes in and you watch it. But what computation on TV means what that actually opens up, not just in gaming, but beyond is that how the end user, the lean back model of looking at a TV interacts with the TV and what they can get out of it is fundamentally changing.

It’s no longer to channel up channel down volume up volume down. It’s about having an interactive panel in your house in which you can actually get real time data of the world, including gaming. So adding computation to TV is what is actually driving a whole new revolution. Adding streaming to TV is what driving a whole new revolution.

And I think that’s actually the biggest acceleration that we’ve seen. The medium of TV since its inception and that’s happened in the last 10 years. So it’s pretty incredible to see what opportunities are gonna open up there.

Geof: And Tetsuya have given the history of Sony and innovating about both game consoles and TVs. I’m sure you have a perspective

Tetsuya: The reality for most TV users is that more events are reliable for delivering high fidelity. Low relationship experiences. If we want global TV content, we may need to balance content streaming with more traditional delivery. On top of that TVs are becoming more capable on their own with powerful processors combined with new sensors. such as a camera. We can start to imagine the TV being able to deliver experience, which rely on immediate environmental input. And couldn’t come from cloud TV is moving from industrial specific commodity to OEM specific. We focus on giving users, digital immersive experience on TV and many key features such as picture quality. Enhancement and interactivity are ongoing planning and developing to the doors. We need more. Computing power but also need system level performance efficiency.

Geof: So it seems clear that the market opportunity’s high with so many people having smart TVs. Maybe we could talk next a bit more about what we actually mean by gaming on TV. Maybe you could tell us about how consumers are influencing this market and maybe Ralph could kick us off on.

Ralph: First of all, we already went a little bit over the statistic of worldwide gaming and people are actually already gaming on TVs today. So the question becomes what is changing and, in Pablo’s note of the 3 billion gamers, how are the consumption patterns for that?

I actually think that we’re still at the beginning of that. So imagine what I just mentioned on the remote the channel up and channel down button that remote that you unpack together with the rest of the TV is you install that unwieldy giant box in your house hasn’t changed yet to how you can effectively interact with the television.

It’s. Mostly to this date about channel selection. So even now we’re talking about two dimensional inputs that allow you to do selection on that TV that is packed in with the box. But if I wanna, an effective gaming experience that also takes into account the medium. , we’re gonna have to see change in control and how it’s controlled and also changing content specifically for, a larger scale audience of 3 billion people.

Not everybody is going to will want to learn how a controller fully works and wants a much simpler interactivity. A TV is a shared screen. A TV is a monitor is a singular screen, but a TV for, in a somebody’s living room is something you look at together. So what type of content do people interact with that has value that multiple users can interact with at the same time?

This is where I was Actually really happy to hear Testuya speak about sensors, because how do you interact when it’s a group of people in the, in a room that interact with the television, not one person holding the remote control is going to be deeply important.

Now I also think that what we call gaming can change. Linear gaming experience or multiplier gaming experiences, they are today are suitable for an audience, but not all audiences. And I think what we’re going to see is an expansion of content being made. Take care of tra traditional game show.. Why wouldn’t you participate in the game show and why wouldn’t that be a more attractive means for people to actually interact with their television? But at similar notes, I also think that we have to be mindful that you can bring content to a television. It doesn’t mean it’s being consumed and we’re gonna still have to learn how that works as an interactive medium. That is the default in people’s homes. A lot of first screen experience in the world right now are being had on mobile phones because they’re price sensitive. They’re, in parts of the world where you might not actually have a TV in the home.

So that audience of people who will consume meaningful, interactive content on their TV is going to change. And that’s where the content needs to follow. Content is always king. So content is always what you switch your TV on for. That’s where it starts. That’s where my thoughts are in most of these processes.

When we think about why would somebody get it? And then it’s about what’s the content and how does it become valuable..

Geof: And how does that sound to you at Tetsuya?

Tetsuya: Yeah, exactly users or TV are getting used to bring game or their TV. And latency or performance is kind, getting more important.

Geof: One of the things that I would imagine as this evolves is that the business model that supports it is also gonna evolve. Pablo if you could kick us off.

Pablo: I think business models is is a really complicated area and gaming has very many different business models. We in arm, we really focus more on enabling those business models that actually how they operate in the industry. I think what for me is interesting. Is that those go-to-market strategies are very depending, very variable, depending on where you are in the world and what kind of market segment you’re targeting.

For example, we are seeing. People in countries like China taking very different approaches to the approach that you would normally take in other parts of the world, in, in the us or in Europe. And we are seeing a lot of interest, for example, in running mobile or Android games on TVs natively because that’s the experience that most of the users have in those in those households, people over there.

Play their games on mobile first. And then when they get home, they want to continue playing the same mobile titles. So for TV vendors offering a native gaming experience of the TV is very attractive. Maybe that same approach works not so well in other parts of the world. and that’s absolutely fine.

Then you have to think about how people engage with their TVs and with their games in in different parts of the world, what are they looking for when they go onto their. Maybe their mobile initially, or the console, but then eventually how, when they go back to their TV, switch it on.

And as Rob said, they have a real control in their hands, which may look like a TV controller or may look like something completely different. But they want to continue to be entertained. So how are you going to realize that opportunity. For us as I say for us, what’s important is that we provide the means the technology to make those business case, the business models, possible.,

Geof: And Ralph, do you have a view on the evolution of the business model?

Ralph: I think some of these are already established markets, so I’m gonna pull it a little bit beyond just games, because I think if we look at the gaming market as a whole, and we look at a subsequent growth over the last 10 years, it’s been massive. The role has seen more gamers and it’s not because.

Massively the market of people who want to play has evolved, but it’s actually the people that can play that has evolved, it’s the accessibility of devices that enable that it’s the mobile revolution that we all went through. And this is where Pablo kind of a. alluded to you wanna play the same game?

I would argue that you don’t necessarily can reach the same audience with the same game on a different screen also because of how distribution channel works, how game developers have to be able to make money on that and the ability for, the creators to engage with an audience. now that’s where I’ll pull it beyond games.

I think part of the gaming technology what realtime 3d really brings the ability to compute, but also to render and what it means for creators. I we’ve seen at Unity Evolution of the medium already by seeing realtime overlays. On top of sports. We have our own meta cost solution, which is in partnership with some of the UFC folks in which, you can now move cameras into places that a real camera can’t even fit in between two fighters.

And I think that’s where the value for content consumer is there to be able to want to interact with it. But of course, then you hit the fact that every league has its rights. And, we’ve seen that throughout the industry of how that works. When you have the intersection of sports and games that can be, get really complicated.

I think the same is happening in games. On itself. We see a consolidation of distribution channels. It’s really hard to break in unless you actually have value in the marketplace that offers to the consumers. This is where I think we’ll find new value on a new medium, and, don’t let me misframe gaming on TV has been around for a long time. We’re just talking about having more users, having that capability, probably with lesser compute. The experience is not gonna be the same probably with a different gaming experience that is more accessible and isn’t, 70 hours that goes into an an RPG game.

So I think we’ll see an evolution there. Distribution bad channels are inherited to the business model. If there’s no distribution channel, the creator can’t make money. So that’s deeply important. I think very often when we talk about technology first, so we talk about the innovation of being able to render on the TV.

We might forget what the consumer offering there is. They’re there to be entertained, to be informed, to have a relaxing moment or a really engaging moment with a sports match. These are the things that will drive consumption. These will drive business model.

Geof: And it sounds like you’re moving towards more, what it would be like if you were at the game in person, but with other things overlaid on top of that, that you wouldn’t get in either experience.

Ralph: There’s two experience I would want to share with that. What’s gaming and what’s entertaining, right? It’s gaming and entertainment. You take an experience like watching a sports match and being able to follow and track your favorite player and that person stats on screen. And you’re able to actually augment a view that the TV has is one.

The other is, the evolution of gaming that is actually somewhat of a regression in gaming. If you think about some of these choose your own adventure stories that you can now play on popular. I know streaming channels. How cool would it be if the character in one of these shows wasn’t a predefined actor, but somebody who matches, your version of the avatar of the protagonist in the story, and you can make those decisions.

I think that’s the, actually the evolution of medium that’s going to happen and real time is what driving. The computation and the devices there creators actually finding the opportunity is what’s going to be driving that. So I’m really excited to see what that future holds for us.

Pablo: And I think what’s really interesting as well there, Ralph is if you think about this content delivery as a complete end to end solution, right?

If you think about how the creator of that content. Can think about what the consumption is gonna look like, what the capabilities are throughout the chain from, as you say the tools, the real time rendering tools that you have on the content creation side, and also the compute capabilities you have at the other end in the TV, then you can start to imagine what what experiences you can, the kind of experiences you can build are completely different. And taking the whole pipeline into consideration enables you to do that.

Geof: My next question is around future devices and this one is for Tetsuya. We spoke about technology earlier and how televisions have evolved to become high performance computers. Where else do you see the once-humble TV going?

Tetsuya: So in addition to. Edge side and cloud architecture is more important. So the content itself getting so various and evolution high, getting higher and higher. Device side like a TV is needed to improve the performance. But on the other hand, so the architecture between rendering and display side need to be more performance and latency is needed to improve.

So the responsibility of crowd and TV device. Need to be considered some vendor already developed about cloud-side rendering. Device side also need to be capable of that.

Geof: That’s exciting. I’m wondering what challenges to getting there.

Tetsuya: Maybe more high performers are rendering power is needed, but in addition to that interactive, not only downstream, but also the upstream side of. Transmitting information from the sensor or recognition on device side.

Geof: Ralph, do you have anything you wanna add to that?

Ralph: Yeah, I’d like to add to that. I think you. Time and time again what we see in the evolution of capabilities and compute is being that it will influence the experiential. So there will be a range of capabilities on the device that enables certain experience to be had. And what type of experience you will have, will be based on, the capabilities of the device. This is you know, the history of gaming on itself how great the experience is going to be is always the thing that is advertised when a new generation of a gaming device comes out.

But I think on phones over the last 10 years, what we’ve seen is an evolution where that becomes less important. And that it is a set of content you can consume at a certain quality and that the choice for quality is going to be on how new your device is going to be or how capable the device is. Which of course from a consumer standpoint, influences their decision quite heavily also with their ability to have access to devices like that.

So I think we’re going to see is not disimilar to, for example, what’s been happening in automotive. what kind of experience do I have as a driver in the car is much more getting much more prevalent and important from a view of, my car has to have interactivity with my phone built in that’s not, an unreasonable ask anymore.

That is a baseline requirement. And I think the same is gonna happen with how we look at devices in our home. They connect, they interact. I can interact with my phone and my TV. My computation of my TV might meet much better than the computation that I get on, my home assistant, but they interact with one another.

So I think that tie in effect is also going to play in quite heavily. That will then dictate your overall experience of content consumption in the home where a TV still is. I’m skimping here on the industrial applications, but I think that’s going to heavily influence how people look at what TV they wanna buy or what set up box they add to it, which is what we’ve seen over the last five years.

The highest quality experience or. The experience that is maybe more cost effective, it changes how you, the content that you can perceive that you can receive and that you can consume in

Geof: Pablo, where do you see the challenges in getting to this kind of a future..

Pablo: Yeah. I agree with Kimura-san and with with Ralph, I think when I look, as Ralph said at the beginning, the last 10 years we’ve seen a significant transformation of TV, the TV space and we in arm notice that the compute performance that we have now is probably 10 or 20 times faster than it was 10 years ago on those very early smart TV.

but that’s still a long way to go from the kind of experience you have on premium mobile devices today. And and then you have to add on top all the sensors cameras and so on that you could potentially build on top of TVs today. The TV screen. It’s probably the only screen you have in the house that doesn’t have a camera attached to it, which is surprising.

But maybe not for long. So when you start to think about the possibilities that that those sort of interactions or those sort of integrations will offer you start to get a glimpse of the possibilities. I say still some way to go I love to, to work with our colleagues here at unity and Sony create a bit more excitement in the industry, build the, the, these new technologies, these new experiences together. It’s it is one step one step at a time, but we are getting there.

Geof: That’s great. And we’ve had a good discussion about the, all of the technology involved, but what about the users of these TVs? Often it’s consumers that drive the popularity of a technology or the content that’s consumed on it.

So what role do they have to play in all of this? And maybe Pablo, you could kick us off.

Pablo: Yeah, of course. Consumers always have the last word and no one knows how some of these new ideas are going to land with with consumers. I think we were. Talking a minute ago about, adding cameras to TVs, or, adding new sensors or these are technologies that consumers may be a bit reluctant to initially at least in, in some parts of the world.

So I think we are all going to go through a phase of trying to communicate, the benefits of these, some of these new technologies. And it’s gonna be the industry as a whole need to Engage with consumers and explain the set of things we’re trying to build and how it is different from what they’ve been able to experience before, either on the TV or on a different set of set of devices.

So it’s, it is Consumer expectation is always very high and is getting higher. Devices are and experiences are incredibly complex already. So if we want to build on top of that and create even more immersive experiences, more compelling stories and more, more entertaining adventures, then we are gonna have to communicate what it is that we’re trying to build together..

Geof: How about you, Ralph, do you have some thoughts on the consumer’s role in this?

Ralph: I think as we talk about it, there’s an evolving medium. One of the most important metrics that I think, we should all take into account is time to content, right? Whether it’s entertainment and it’s interactive, or it’s a stream, I think the way consumers make choices, depends on the settings and what it offers, but when the medium is evolving, then you also start offering new things and as we talked about earlier being able to game on your TV is one, one thing, but then to actually be the audience that wants to have that experience is another. And I think that’s where the content will matter to every consumer. It’s going to be many different. Pieces types of interaction, augmentation of the TV.

Pablo earlier mentioned cameras. I think we’re gonna see a very steep curve before, consumers accept cameras in their TVs because, we were in a world where cameras are becoming more invasive and specifically in a private space, like your own living room where this camera can be used a consumer would want to have a real use case for the camera to be there now. An advertiser might see immediate value. How many eyeballs are watching this show right now? And who’s actually paying attention. But that’s not what a cons gets a consumer over the threshold of going there. What gets a consumer there is why does that camera add value for me?

So it might be video conferencing. It might be that the show is able to see me and take my avatar into the show. It might be that the camera allows me to interact by waving at the TV and saying pause by holding my hand up.. But I think that, that is the first choice that there’s going to be a lot of resistance crying around these types of things.

If there isn’t actual consumer value. And we only think about it from a content consumption value. And that’s where I’m saying content is king. Content being king specifically when we’re talking about an interactive, medium means that for the large screen and the home for an audience of billions, we still have a lot of ways to figure out what the value then really is.

Real time overlays, real time 3d on sports shows. Real time replays, where you can put your own camera somewhere else, the ability to decide the interactivity or, one of the things that I’ve noticed in my kids a lot who are some of the primary consumers of some of the streaming channels we have is the interactivity with the show where they can find themselves and mirror themselves in the show where, it’s real time rendered.

Now it’s still TV consumption, but it connects to the actual. Person watching it. That’s where value starts getting built. And that’s how people make decisions, right? If I’m gonna go into a store, why would I buy the TV with the camera versus the one without needs to have a proper value proposition of the why towards the user, not for the industry.

Geof: And Tetsuya what role do you see consumers having in this amazing future?

Tetsuya: About consumers. So I suppose so not only just watching content itself, but since a game or content becoming more interactive, so difference between creators and consumers getting closer, so sometimes uh, consumer could join the create content itself. So because of current recently game have some creative mode. If the consumer have some way to change the content itself, users as a creator is our, my, another vision for the future..

Geof: Thank you for that. And thanks so much to all of you to Pablo, Ralph and Tetsuya for giving me so much to think about when I’m sitting on the couch in front of my TV, deciding what I wanna watch or play next.

But I have a feeling that after this conversation, I may need to get myself a new TV to do it. So thanks to everyone for joining us today. We hope you enjoyed it. And we look forward to joining you again soon on the next episode of Arm Viewpoints.