Case Study: Deliveing an immersive gaming experience for Intel based hybrid devices

Abstract

Tencent wanted to give gamers the best experience on Intel® Ultrabook™ and 2 in 1 systems. Legend of Xuan Yuan was already a successful game, but these systems provided Tencent with a new opportunity. Many systems currently provide 2 in 1 usage, meaning they can be handled as a traditional laptop or a tablet. Tencent worked with Intel engineers to detect the laptop and tablet modes to change the games state. They updated the UI to support touch, which has become one of the most essential and exciting features on tablets. Finally, the systems sensors allowed new gameplay by including "shake" to enable a special action in the game.

Introducing the first touch 3D MMOPRG for the Chinese market

Tencent is the biggest game developer in China. With a growing number of 2 in 1 systems in the Chinese market, Tencent wanted to give their players a unique experience. After two years in the market, Legend of Xuan Yuan was already a popular title. The availability of Ultrabooks and 2 in 1 systems made it the right time to add touch and accelerometer support to the game.  Although 3D MMORPGs are very popular in China, none of them supported touch before Legend of Xuan Yuan. Tencent had a chance to innovate, but there was also risk – would the changes be successful? This case study illustrates how, working with Intel engineers, Tencent changed the game to play well on 2 in 1 systems and Ultrabooks running Windows* 8.

Legend of Xuan Yuan needs two different UIs for tablet and laptop modes. On 2 in 1 systems, the game detects when the system is used as a laptop versus a tablet. The game uses keyboard and mouse when the system is in laptop mode. When its used as a tablet, the game switches to a touch-only UI. Tencent wanted an effortless transition between the traditional laptop mode and touch gameplay. The player has a seamless experience because the UI changes automatically to suit each mode. In this case study, well look at how to detect the mode of a 2 in 1 system and change the UI based on that mode.

Converting an existing user interface to touch can be difficult. Its especially hard for games with rich UIs that rely on left-click, right-click, and multiple command keys. Theres no single formula for adapting this kind of UI. It requires great care to deliver smooth and satisfying gameplay via touch. Because the game had an existing installed base, the team was careful to make the smallest changes possible and not alienate existing players. Well review the UI design.

Since these systems include an accelerometer, Tencent also added support for a "super-kill" attack against opponents when you shake the system during gameplay.

Changing game mode to match the 2 in 1 state

Legend of Xuan Yuan has two UI personalities and dynamically changes the UI based on the state of a 2 in 1 system. When the system is in laptop mode, Legend of Xuan Yuan plays as it always has with keyboard and mouse input. When the system is in tablet mode, the player uses touch input. How does this work?

Heres how we did it: Detecting 2 in 1 state changes and changing UI mode

Legend of Xuan Yuan listens for the WM_SETTINGCHANGE message. This message notifies apps when the system changes state. The WM_SETTINGCHANGE message comes with its LPARAM pointing to a string with a value of "ConvertibleSlateMode" when the 2 in 1 state changes. A call to GetSystemMetrics(SM_CONVERTIBLESLATEMODE) reveals the current state.

When the game is in tablet mode, it displays an overlay UI with touch buttons for the various UI actions. It hides the overlay UI in laptop mode.

Legend of Xuan Yuan uses detection code like this:

Figure 1: Code to detect system setting change and check system mode

For more details, check the basic 2 in 1 aware sample.

This technique must be enabled by the system OEM, with a supporting driver installed, in order to work. In case its not properly enabled on some systems, we included a manual way to change the UI configuration. A later section describes how the UI works.

Now its your turn: Detect if the system is used as a laptop or tablet

How can you detect the system state for your game and how should it change its UI? To play best on 2 in 1 systems, your game should have dual personalities and dynamically change its UI based on the state of the system.

First plan a UI for both laptop and tablet modes. Consider how the system might be held or placed. Pick the UI interactions that work best for your players. As you design the touch interface, you should reserve more screen area for your touch controls than you typically need for mouse buttons. Otherwise, players will struggle to reliably press the touch controls.

Touch interactions are often slower than keyboard and mouse, so keep that in mind too. Game menus also need both a keyboard plus mouse UI and a touch UI.

Its a good idea to check the system state at startup with GetSystemMetrics and set your UI accordingly. Remember that not all systems will correctly report their state or notify your game of state changes, so choose a default startup state for your games UI in case the state isnt detected.

Listen for the WM_SETTINGCHANGE message once the game is running. When the message arrives, check its contents for an LPARAM pointing to a string with a value of "ConvertibleSlateMode". That value indicates that the game should call GetSystemMetrics(SM_CONVERTIBLESLATEMODE) and check if the UI should change.

Detection may not be conclusive because all systems may not correctly report state changes. Your game should probably default to laptop mode if the detection doesnt give certain results. It should definitely include a way to manually change the UI between keyboard/mouse mode and tablet mode.

For a complete sample that detects system state and changes its UI, look at the 2 in 1 aware sample. To see a more complex sample that detects docking, screen orientation, and more, check the detecting state sample.

Deciding on a touch message type

Youll need to decide which Window message type to support before you can add touch support to an existing application. Choose one of the three different sets of Window messages: WM_POINTER, WM_GESTURE, or WM_TOUCH. Well walk through the decision process used for Legend of Xuan Yuan and examine ways you can do the same for your game.

How we did it: Comparing touch message types

Touch support is at the center of the new version of Legend of Xuan Yuan. When players use the touch screen, they see a new UI with a set of touch controls on screen.

WM_POINTER is the easiest message type to code, and it supports a rich set of gestures. WM_POINTER only runs on Windows 8 and beyond. Tencent wanted to support a large installed base of Windows 7 players, so WM_POINTER was not the right choice.

Before we discuss the remaining message types, lets review the key UI elements for Legend of Xuan Yuan. The games touch UI uses on-screen button controls. These controls can be used for movement and actions at the same time. The movement and action controls are on opposite sides of the screen, for use with two hands. These controls are in the bottom corners of the screen. Theres also an icon near the top of the screen to bring up a cascading menu for more complex UI elements. Well discuss the design of the UI later, but this gives us a good idea how the UI elements must work.

Figure 2: On-screen touch overlay UI, in left and right bottom corners

The game must recognize simultaneous points of contact from different parts of the screen. Because multiple touches must work at the same time, we refer to this as multi-touch.

Now that we understand the main parts of the multi-touch UI, we can compare the remaining touch message types: WM_GESTURE and WM_TOUCH. The easiest one to code is WM_GESTURE, which has simple support for typical gestures like a two-finger pinch (zoom) and finger swipe (pan). This message type hides some of the detail of touch interaction and presents your code with a complete gesture once the gesture is done. Simple touch events are still sent to your game as mouse messages. This means a typical touch interface could be implemented using mouse messages for simple touch events plus WM_GESTURE for complex gestures.

The gestures supported by WM_GESTURE can only include one set of related touch points. This makes it difficult to support gestures from this kind of multi-touch UI where the player touches the screen in different places. WM_GESTURE is a poor choice for this game.

WM_TOUCH is the lowest-level touch message type. It gives complete access to all touch events (e.g., "finger down"). WM_TOUCH requires you to do more work than the other message types since you must write code to represent all high-level touch events and gestures out of low-level touch messages. In spite of the extra work required, WM_TOUCH was the clear choice for Legend of Xuan Yuan. WM_TOUCH gave complete control over all touch interaction including multi-touch.

When theres a physical touch on the screen, the system sends WM_TOUCH messages to the game. The game also receives a mouse click message at the same time. This makes it possible for apps without full touch support to behave properly with touch. Because these two messages of different types describe the same physical event, this can complicate the message handling code. Legend of Xuan Yuan uses mouse-click messages where possible and discards duplicate messages.

Your turn: Choosing the right touch message type for your game

WM_POINTER is a great option if your game will only be used on Windows 8. If you need backward compatibility, look at both WM_GESTURE and WM_TOUCH messages.

Consider your UI design as you compare the message types. If your UI relies heavily on gestures, and you can easily write mouse-click handlers for the non-gesture single touch events, then WM_GESTURE is probably right for your game. Otherwise, use WM_TOUCH. Most games with a full-featured UI use WM_TOUCH, especially when they have multiple controls that players will touch at the same time.

When you evaluate the touch messages, dont forget the menu system. Remember also to discard extra messages that arrive as mouse clicks.

To learn more about the tradeoffs between the three message types, see this article. For more detail on choosing between the backwards-compatible WM_TOUCH and WM_GESTURE message types, see Touch Samples.

Adapting the UI to use touch

Adapting an existing game UI to touch can be complex, and theres no single formula for how to do it well.

How we did it: A new touch UI

The keyboard and mouse UI is familiar. It uses the W, A, S, D keys to move the character. Customizable action keys on the bottom of the screen and shortcut keys 1-9 hold potions, attack skills, and open richer UI elements. These UI elements include inventory, skill tree, task, and map. Right-click selects the characters weapon and armor or opens a treasure box.

The touch screen is available at all times, but the touch UI is hidden by default during keyboard and mouse gameplay. A touch button is visible on-screen in this mode.

Figure 3: Pressing this touch button in this mode brings up the touch UI

If the player switches the system to tablet mode or touches this button, the full touch UI appears on-screen.

How we did it: Elements of the touch UI

In tablet mode, the player usually holds the system with both hands. The UI layout uses both thumbs to minimize any grip changes. Move and attack actions are grouped for easy access by the players left and right thumbs.

First, we designed a wheel control to move the character. The wheel is an overlay on the left side of the screen. This is similar to game controllers, and this familiar use and placement makes it easy to use. The players left thumb will usually be in constant contact with the screen. As they slide their thumb around, the character moves on-screen in the direction of the players thumb.

The regular in-game action bar is at the bottom of the screen, but that doesnt work well for thumb use. We added a group of 4 large action buttons in the bottom right corner where the players right thumb can easily reach them. The player can configure these to trigger their most frequently-used actions by dragging attack skills or potions to each button.

The player must target an enemy before attacking. With the keyboard/mouse interface, left-click targets a single enemy and TAB targets the next enemy within attack range. In touch mode theres a large button to target the next close enemy. The player can also touch an enemy to target them directly, but thats not common since it disrupts the players grip on the tablet.

The keyboard and mouse UI uses right-click to open a treasure box, equip a weapon or armor, or drink potions. Tap and hold is the best touch replacement for right-click, so it replaces right-click for the touch UI.

With the keyboard/mouse UI, theres a small icon on-screen to open the cascaded windows. This doesnt work well for touch since the icons are too small. The touch UI includes an icon on-screen to bring up the rest of the UI elements through a cascading set of icons. These icons bring up more complex parts of the UI like the inventory bag, skill tree, tasks, etc. There is also an option to toggle the UI between the keyboard/mouse and the touch overlay. This gives the player an easy way to change between the two UIs.

Figure 4: Full touch UI with movement wheel, action and target buttons, and the cascading UI displayed

Heres the full touch UI with the cascading icons open.

How we did it: Message handling for the touch UI

How does the message handling work? It varies for different parts of the UI. Both WM_TOUCH and mouse messages are used. The action, targeting, and cascading UI buttons all use mouse click messages. The movement wheel and main part of the game screen use WM_TOUCH messages.

Typical gameplay involves continuous touching on the movement wheel control, with repeated use of the enemy selection and skill attack buttons. This means that good multi-touch support is essential. Luckily, WM_TOUCH has good support for multi-touch.

When theres a WM_TOUCH message, the game saves some context. It compares this touch with other recent WM_TOUCH messages, checks how long the current sequence of touches has been held, and looks for the location of the touch.

If the WM_TOUCH message was on or near the movement wheel, the code checks the location of the touch relative to the center of the wheel and the previous touch. If the touch was close to a previous touch and this current gesture started on the wheel, the game moves the character in the desired direction. During development, this required some careful configuration to detect the difference between long continuous touches on the movement wheel and other touches on the main part of the screen.

If a WM_TOUCH message is on the screen away from the other controls, then it might be part of a gesture like zoom or pan, or it may be part of a tap-and-hold. WM_TOUCH messages are compared with previous ones to decide which action to take. If its close enough to the first and has been held for longer than 0.2 seconds, its treated as a tap-and-hold. Otherwise, its a gesture so the screen is adjusted to match.

The system also automatically generates mouse messages for all touch messages. Each mouse message includes extra information detailing where it came from. The GetMessageExtraInfo call identifies the difference.

Figure 5: Check if mouse messages came from touch screen

When a mouse message was generated from the touch screen and the game has already handled the physical touch via WM_TOUCH, the game discards the mouse message.

If a touch message is on one of the other controls, then it is discarded and the mouse message is used instead.

With all UI elements in place, the game plays well with a touch screen.

This article shows another example of adapting a complex UI to touch in Wargame: European Escalation: https://software.intel.com/en-us/articles/wargame-european-escalation-performance-and-touch-case-study

Your turn: Building your touch UI

Before you build a touch UI for your game, imagine all the actions a player might take. Then think about how they might be done with touch (or other sensors like the accelerometer). Pay special attention to the differences between tap and click, continuous actions like press-and-hold, and gestures like drag.

Decide how the player will do all of these actions with touch and where the visible controls should be. For any on-screen controls or cascading menus, ensure they are big enough to use with a fingertip or thumb. Think about how your typical player will hold the system, and design your UI for easy touch access with a typical grip.

Now that you have the UI planned, use the simplest message for the job. Identify when a touch hits each control. Plan which message type to use for those controls (mouse or touch) and discard duplicate messages.

For touch messages, save the context of the touch message. Location, control, and timing will all be useful when you need to compose gestures out of multiple touch messages. Think about the parts of your gameplay that require continuous touch contact. Carefully test this during development to make sure that your game works well with typical variations in gestures. Check a variety of gesture directions, touch locations, proximity to previous touches, and touch durations.

Start the UI in whatever mode matches the systems current state. Switch the UI between touch and keyboard/mouse whenever the system state changes. Finally, remember to include a manual way to force the UI change in case the system isnt configured to notify you properly.

For more tips on designing your touch UI, see the Ultrabook and tablet Windows touch developer guide.

Sensors

Ultrabook and 2 in 1 systems include sensors like gyroscope, accelerometer, GPS, etc. Its possible to enhance the gameplay experience with them.

How we did it: Shake for a special action

Legend of Xuan Yuan uses the accelerometer to detect when the player shakes the system. The player accumulates energy during gameplay, then releases it during a super kill attack. The player can shake the system to trigger the super kill, which attacks nearby enemies for 10-20 seconds.

We tested some different shake actions to measure typical values from the accelerometer:

Figure 6: Four shake actions, showing intensity and duration in 3 dimensions

Any acceleration over 1.6 on one axis is a shake. We could also use the sum of the absolute values of acceleration on each axis.

Because these are real-world events, the data will be noisy and different each time. The values include both long and short shakes. While most of our test shakes gave a single peak value, one of them had several near-peak values. This game uses any shake over 1.6 in any direction on any axis. Multiple shakes within 1.5 seconds are grouped together as one.

With this code in the game, any shake action will unleash a super kill action.

Your turn: Use the systems sensors

Ultrabook and 2 in 1 systems contain a number of sensors. Be creative, and think of ways you might use each of them to enhance your gameplay.

Whichever sensor(s) you use, calibrate them to see how they react in real-world conditions. Consider the different typical conditions your players will encounter.

Summary

Weve shown how to adapt an existing game to detect the state (laptop or tablet) of a 2 in 1 system. We also demonstrated how the UI can support touch and how to switch between UIs based on the 2 in 1 system state. Along with the accelerometer to trigger a unique action in the game, these give a compelling game experience.

Tencent took a risk by introducing the first Chinese MMORPG to support touch gameplay. The risk has paid off! Legend of Xuan Yuan plays great on laptops, tablets, and 2 in 1 systems. We hope you have similar success with your game!

Source: https://software.intel.com/en-us/articles/legend-of-xuan-yuan-case-study-get-the-best-gameplay-with-2-in-1-state-touch-and

For more such windows resources and tools from Intel, please visit the Intel® Developer Zone

Follow Us

Promotion