With the Microsoft announcement of the Surface Neo and Surface Duo late last year, it is clear that the next wave of devices will be vastly different than the standard computers we are used to. With these changes in devices comes some very interesting changes for web development.

Now that we have started the new year, Microsoft has followed up these hardware announcements with early details on what developers can expect and how to begin preparing applications for the coming change. Although the majority of information on these development changes are targeted towards Win32 and UWP applications, Microsoft has put together some proposed standards and specifications for new CSS and Javascript APIs to take advantage of this new device category.

Let's explore these dual screen application concepts, take a high level look at these proposed API additions, and make a small demo of how a responsive site handles dual-screen devices.

Table of Contents

  1. Introduction to Dual Screen Devices
  2. Basics of Dual Screen Applications
  3. Dual-screen devices and the web
  4. Theoretical Implementation
  5. Final Thoughts

Introduction to Dual Screen Devices

If you follow any computer or phone news, you will no doubt see various devices that consist of a folding screen or two screens connected by a hinge. This concept is not completely new to the industry but is, for the first time, going to become a production product instead of a prototype. This push to release production dual-screen devices for personal computing is being championed by Microsoft's Surface Neo and Duo as well as various other OEM devices.

Basics of Dual Screen Applications

Let's discuss some of the key concepts and challenges for dual-screen applications. These details are a basic summary of Microsoft's Introduction to dual-screen devices along with some additional thoughts and extrapolation.

Device folding and orientation

The fundamental key concept of dual-screen devices is their ability to fold, flip, and rotate. Microsoft calls these changes "postures".

These "postures" are an extension of the changes that have been introduced by smart phones and tablets. This evolution of layout possibilities are likely to make UI design more interesting and complex, but also provide an amazing and more intuitive experience to users.

Consider how fundamentally different UI design has become since the explosion of web for mobile devices. Consumers are now expecting sites to be mobile friendly, and developers have stepped up with new patterns and practices that take full advantage of features, like touch and device rotation, as well as overcoming challenges like smaller screen size. With this in mind, let's look at some of the challenges with dual-screen development.

The Seam

Probably the most difficult UI challenge with this new category of devices is dealing with the seam created by the hinge between the two screens. Microsoft's Introduction discusses this challenge in more detail here, but the biggest takeaway (and probably most prevalent issue for current websites in some "postures") is not always being able to use traditional center alignment. Let's take a look at one of Microsoft's examples.

dual-screen devices and seams

Looking at this suggestion it is not hard to think of the vast number of websites that will be far less functional when spanning dual-screen devices.

New Orientations

This challenge will likely end up being a moot point, but the different options of orientation and fold angle between the two screens only adds to the complexities (and possibilities) of UI design. For example, a UI designer may want a site or app that spans two screens to adjust slightly depending on if the user is holding the device like a book, versus laying the device down flat on a table. It is currently unclear how much of the hinge orientation data will be available through APIs, and I may be over complicating  a simple thing, but it is interesting to think about.

Optimize for alternative inputs

With touchscreens becoming commonplace among all types of computing, alternative inputs like touch are becoming more common. Although touch input has become common place thanks to smart phones and tablets, the importance of supporting additional inputs will become more important as dual-screen devices reach consumers.

This is not a comprehensive list of UI design and changes that dual-screen devices brings to the programming world, but it covers the basics. With that out of the way, let's look over the proposed CSS and Javascript API standers.

Dual-screen devices and the web

As of the beginning of February 2020 standards on Dual-screen devices have not been given from ECMA International or the  CSS Working Group, but Microsoft has outlined the following suggestions on Github to start the discussion along with suggested polyfills.

Without going into too much depth, let's look at the basic propositions and imagine some real world application use cases.

New CSS primitives

"spanning" media features

The first proposed CSS change is the introduction of a new Media Query feature for determining if the window is currently spanning across two screens. This spanning media type would allow 3 possible media types: "single-fold-vertical", "single-fold-horizontal", and "none".

Device fold environment variables

In addition to the basic proposed media query, the "fold" environment variable would provide details about the device seam (or fold) using the following variables: "fold-top", "fold-left", "fold-width", "fold-height".

dual-screen css fold variables

New Javascript API, Windows Segment Enumeration

Microsoft's suggestion for Javascript APIs is the Window Segment API. This would be an addition to the standard window interface and would allow the programmer to get information on the available screens using window.getWindowSegments(). This new function would return an array of segment details (1 element for each screen) with a minimum of top and left properties. If the current window does not span more than 1 display, then the array will return a single element. You can read more about this here.

Theoretical Implementation

Now that we have covered the the basics of the proposed API, let's look at some theoretical code. In order to test this we will be using this web based dual-screen emulator and a polyfill by Zouhir. Let's use a basic HTML page and CSS grids to build a dynamic UI that supports desktop, mobile, tablet, and dual-screen devices.

index.html

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <link rel="icon" href="favicon.ico" />
    <title>Dual Screen Example</title>
    <link href="style.css" rel="stylesheet" />
  </head>
  <body>
    <div class="App">
      
        <div class="header">Header</div>
  
        <div class="stories">Story Data</div>
  
        <div class="content">Content</div>
  
        <div class="related">Related</div>
  
      </div>

      <!-- Add the polyfill at the end -->
    <script src="spanning-css-polyfill.js"></script>
  </body>
</html>

style.css

body {
  margin: 0;
  font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Roboto", "Oxygen",
    "Ubuntu", "Cantarell", "Fira Sans", "Droid Sans", "Helvetica Neue",
    sans-serif;
  -webkit-font-smoothing: antialiased;
  -moz-osx-font-smoothing: grayscale;
}

.App {
  display: grid;
  grid-template-columns: 10vw 67vw 20vw;
  grid-template-rows: 5vh 60vh 33vh;
  grid-template-areas:
    "header header header"
    "stories content related"
    "stories content .";
  column-gap: 1vw;
  row-gap: 1vh;
}

.header {
  grid-area: header;
  background-color: lime;
}

.stories {
  grid-area: stories;
  background-color: maroon;
}

.content {
  grid-area: content;
  background-color: mediumorchid;
  overflow-y: auto;
  min-height: 50vh;
}

.related {
  grid-area: related;
  background-color: mediumslateblue;
}

This gives us a very basic layout with color that helps indicate where individual components are displayed.

example desktop layout

Now that we have the desktop layout, we need to make this responsive. Let's add our media queries and changes for phones and tablets first.

style.css

/*
phone layout portrait
*/
@media screen and (max-device-width: 480px) and (orientation: portrait) {
  .App {
    display: grid;
    grid-template-columns: 100vw;
    grid-template-rows: 10vh 10vh auto 10vh;
    grid-template-areas:
      "header"
      "stories"
      "content"
      "related";
    column-gap: 1vw;
    row-gap: 1vh;
  }
}

/*
tablet layout portrait
*/
@media screen and (min-device-width: 480px) and (max-device-width: 1200px) and (orientation: portrait) {
  .App {
    display: grid;
    grid-template-columns: 25vw auto;
    grid-template-rows: 10vh 80vh 8vh;
    grid-template-areas:
      "header header"
      "stories content"
      "stories related";
    column-gap: 1vw;
    row-gap: 1vh;
    font-size: 2.4vh;
  }
}

With these changes we can now see the layout update with different device sizes.

responsive layout example

Phone and Tablet devices are now complete, so let's tackle the dual-screen devices using the proposed Microsoft standards we discussed earlier. We can accomplish this by adding media queries for dual-screen devices. Notice that the sizing of columns and rows have changed to use the proposed fold-left, fold-top, fold-width, and fold-height properties. This layout does require more calculations because of the fold.

style.css

/*
  dual screen layout landscape
  */
@media (spanning: single-fold-vertical) {
  .App {
    display: grid;
    grid-template-columns: calc(env(fold-left) - 1vw) env(fold-width) calc(
        100vw - env(fold-left) - env(fold-width) - 1vw
      );
    grid-template-rows: 5vh 60vh 33vh;
    grid-template-areas:
      "header header header"
      "stories . content"
      "related . content";
    column-gap: 1vw;
    row-gap: 1vh;
  }
}

/*
  dual screen layout portrait
  */
@media (spanning: single-fold-horizontal) {
  .App {
    display: grid;
    grid-template-columns: 60vw 39vw;
    grid-template-rows:
      9vh
      calc(env(fold-top) - 10vh - 2vh)
      env(fold-height)
      calc(99vh - env(fold-top) - env(fold-height) - 2vh);
    grid-template-areas:
      "header header"
      "stories related"
      ". ."
      "content content";
    column-gap: 1vw;
    row-gap: 1vh;
  }
}

With the above changes we can now see the layout change behavior in the emulator.

dual-screen responsive example

You can play around with the emulator here. The basic site is accessible here and code is available on Github.

This is a very simple example and does not touch the Javascript API suggestions, but I think provides a starting point for conversation related to how these new device form factors may influence the work of web designers.

Final Thoughts

I think dual-screen devices provide some interesting options for web UI and will be an important factor in the developing Web Apps and Sites in the future. In my theoretical implementation, there were some challenges that I was not expecting.

  • The area of the fold is NOT removed from the pixel counts. In the current suggested spec, the screen view size includes the virtual pixel space that would be taken up in the fold. For example, if each screen is 500px and the fold is "30px" wide, 100vw would be 1030px. Because of this, the css classes for the dual-screen display required a lot of calculation to prevent elements from being obscured by the fold.
  • When the device is rotated from landscape to vertical, the start position of the fold changes from fold-left to fold-top. This makes sense and is not hard to change. What didn't make initial sense to me was the fold-width and fold-height properties. When in landscape, the fold-height environment variable is equal to the 100vh value, and in portrait the fold-width is equal to 100vw. This seems useless as the key thing you want is something like fold-size, which is just the pixels that the fold would virtually take.

Overall I am looking forward to this new form factor as well as the new UI standards and designs that will emerge from the community.