Assuming Control: my past, our present & the possible futures of game interaction

Assuming Control feat

The current ‘Blogs of the Round Table’ topic on Critical Distance concerns game controls and the ways in which we interact with what we play:

Joysticks. Keyboards and mice. Mashing a controller with your fist. Touching. Poking. Waggling. Wiggling. Moving your head around a virtual reality world. Directing an arc of your own urine. The ways in which we can interact with games have changed from simple electrical switches into much more complex and nuanced forms. We can even adapt and alter controls for people who have difficulty using traditional methods.

Some of these methods work, and some don’t. Most of us we be familiar with complaints about the Wii’s “waggle” controls, the thumb-numbing frustration of virtual buttons on a touchscreen device, or the gyroscopic motions that ruin the 3D bit of the Nintendo 3DS.

How do we move forward with controls in games? Are the old ways the best, or a barrier to entry? Are you looking forward to playing Farmville on the Facebook-ulus Rift?

It’s not a topic that I have historically devoted significant thought toward. But bear with me – I do have thoughts on the matter.

For most of my life well wasted I have played PC games, which has meant that I have largely settled into a pattern with the mouse and keyboard. Of course I have upgraded these peripherals over the years; the Razor Diamondback I use today is a far cry from the primitive mice I first used. Nowadays I use a Microsoft keyboard that calls itself a “Digital Media Pro”; I find using it supports my wrists and typing style well, which helped with the RSI I suffered almost a decade ago (this rather dates my keyboard, but don’t worry – I shake the crumbs out every so often).

The PC’s enjoyed various peripherals too, of course. I know a few people who bought themselves expensive steering wheels for racing games, and I played a lot of space sims in the 90s using cheap QuickShot joysticks; I purchased a more modern though still affordable stick a few years ago so I could revisit Conflict Freespace. And of course since the release of the Xbox 360 its controllers have grown increasingly popular among PC gamers (I’m sure some weird deviants grew to prefer the PS3 pad, too).

That’s not to say I’ve never dabbled in consoles. I owned a NES long before I got my own PC, and that was replaced with a SNES after some years; I remember the hard plastic lumps and their stiff plastic buttons well. Occasionally I’d get to play their SEGA counterparts when visiting friends or staying with a childminder. Later, I experienced jealousy when childhood friends were given Saturns or N64s (jealousy which I turned around when I showed off Quake 2 with my new Orchid Righteous 3D card… but that’s a whole other story). I eventually bought myself a PS2 when I graduated from university, and upon finding myself a job I followed it up with an Xbox, Gamecube and Dreamcast. A few years later the Wii, Xbox 360 and eventually PS3 joined the collection.

With each console added to the line-up I found myself rating the respective merits of each controller, albeit not in tremendous depth. The single-sticked Dreamcast controller seemed to me an intolerable throwback, whilst the Gamecube’s odd layout and configuration seemed like a child’s toy until I learned that it worked surprisingly well – invisibly so – with a lot of the games I liked most on the platform. The weak analogue sticks and unimpressive shoulder buttons of the PS2’s DualShock hindered an otherwise solid controller, for me, and I ended up preferring the original Xbox’s s-controller (the console’s original controller, equivalent in size to the Wii console itself, was of course one of the funniest jokes about Microsoft’s first foray into the console marketplace).

N64 controller

I have never liked these things.

Handhelds, too, had their appeal: my first Gameboy was a red Pocket, and some years later I picked up first a DS, then a DS Lite, and finally a dirt-cheap PSP. The 3DS and the Vita appeal as well, as much for their interaction innovations than for any of the games I could actually play on them. And of course I’ve joined the smartphone generation, playing iOS and Android games on various handheld devices. I even bought myself a Kinect, and although using it in a small British living room is a challenge North American readers may not understand it does get dusted off semi-regularly. I’ve had some genuinely excellent times monkeying about with the voyeuristic little robot.

I’m sharing all this to provide some context around the modes of videogame interaction I’m familiar with, and to give you some perspective on the controller biases that I may possess but be unaware of… as well as those I’ve already expressed. All of the above is what I know: I’m au fait with gesture and voice controls, with wand and light sensor controls, with touchscreen controls, with controllers of various generations, with joysticks and of course with the mouse and keyboard.

I’m weighed down by the burdern of decades of experience with these devices. It is difficult to see beyond the expectation of familiarity.

It strikes me that insofar as the questions “How do we move forward with controls in games? Are the old ways the best, or a barrier to entry?” are concerned, we may be looking at this backwards. In design terms the hardware often comes first, with games subsequently designed within the parameters established by hardware specifications.

What if we inverted this? What if we set out to first design games, and then to identify how best they could be interacted with?

It’s rarely an economically viable proposition outside of research units and groups, of course. The videogame industry is ultimately a grouping of businesses who are out to make money, and finance dictates that the colossal hardware variance that would result from games-first design would be a poor business model with significant upfront and R&D costs. Independents are less beholden to the protective conservatism of businesses, though they too must have their vision limited by what they can realistically afford. For most, designing, prototyping and then mass-producing suitable hardware is an absurd proposition.

But if we are to really answer the question of “where do we go from here?” then it seems logical that we must put the need before the solution; the challenge before the resolution. Ultimately, videogames are a design-led industry and it is from here that such answers must emerge – not the ill-informed whims of a blogger, even if – perhaps especially because – he has experienced a lot of what the last three decades has had to offer.

That said, it is fun to engage in a little speculation. I suspect that the next five years will primarily involve the consolidation of existing technologies.

Gesture control will become more sophisticated as the sensors and software improve: already the Kinect 2 is lightyears ahead of its predecessor and WiSee suggests another route that similar technology might take in future. That said I do suspect it will be a long time before gesture control moves beyond very general sorts of interaction – far from the finesse that sophisticated modern games demand. There is the possibility that sophisticated interactions could be composed from sequences of gestural interactions, but the obvious risk there is that the error rate in gesture recognition is still high enough that individual gestures are often misinterpreted. That does not bode well for complex sequences. So, right now, gestural input in a videogame is typically tied to a very small number of gaming verbs.

Tony Stark gesture controls

CGI gesture/touch controls have cropped up a lot in films since 2002’s Minority Report. The realities are a lot less sophisticated and sexy (and using a Kinect does not make you feel like Tony Stark).

Touchscreens will improve in responsiveness and I am confident that connecting a smartphone to a larger screen will become increasingly popular, but the already-understood problems with touchscreens will persist. Haptic feedback (vibration and similar responses to the touch of a finger) cannot make up for the lack of tactility where a ‘virtual d-pad’ or ‘virtual thumbstick’ is in use, and the control interactions obscure the main feedback medium, i.e. the screen. I would honestly be surprised if touchscreens alone developed much further in terms of their utility as a controller: it’s more likely that we’ll see continued integration with other devices, as with the Wii U. There’s also potential scope for hybridization of non-flat or flexible touchscreens with traditional gaming controllers. For example, a semi-flexible touchscreen could be laid over a simple D-pad, offering physical resistance and movement as a response to the user. This offers clearer feedback on where the player’s digits are and what they’re doing than touchscreen haptic feedback ever could.

The focus of modern controller development appears to be primarily around ergonomics and form factor, alongside refining wireless functionality and durability, and frankly I’m quite happy with that. The controller, like the mouse and keyboard, is one of the oldest and most established forms of game interaction, and I don’t see any imminent need to reinvent these particular wheels.

Voice control will, I think, become more prevalent over the next five years. The trends are already in this direction with the growing sophistication of voice recognition software. There will be great challenges in building games that work with such software, but I suspect it will not be too long before we see games that are to Tom Clancy’s EndWar what Street Fighter IV is to Karate Champ.

As for the mouse and keyboard… well, I use them constantly for many things as well as playing games, so they’re not going anywhere. I laugh in the face of anyone who genuinely believes that touchscreens will replace physical keyboards: see above regarding haptic feedback vs. tactility.

But this is just me. Others have contributed to this round table discussion; see the form below to check out what they’re thrown into the ring. It would also be fascinating to see your thoughts in our comments thread below!

[I originally drafted this piece in late April, before Joel Goodwin wrote ‘Of Mice and Gamepads’. In it he speaks with numerous game developers and designers who are pushing the boundaries in terms of videogame interaction and the future thereof. It’s a fantastic read, particularly as it offers so many perspectives from people who are directly engaging with new interaction technologies – rather more engaged in the realities than my speculation. So go read!]