Accessible JavaScript Gestures

Share on FacebookShare on LinkedInShare on Twitter

One of the problems with creating accessible dynamic HTML applications that work well across multiple browser and AT platforms is the issue of keyboard accessibility on a touch device. The major mobile operating systems like iOS and Android all allow a keyboard to be attached (via Bluetooth or otherwise) and used in conjunction with the other assistive features. In fact on iOS, the keyboard is only really usable as the major input device when you have VoiceOver turned on.

This leaves two problems from an accessibility perspective:

  1. When someone is walking down Michigan Ave. with their white cane in one hand, it is not really practical to expect them to whip out the keyboard and pair it with their iPhone to achieve a task with your web site/application. It is often difficult enough to just free up two hands to be able to make the gestures required to use the device.
  2. iOS, once again disappoints (or maybe “inspires” is the right word) in that the arrow keys do not get delivered to the event handlers.

This means that if you are implementing some complex ARIA widget interaction such as the ARIA authoring guidelines keyboard interaction for a date picker, then you are out of luck.

You have a couple of options for the date picker in particular – you could simply defer to HTML5 input type=”date” and this would probably be my recommendation to you for this particular case, but what happens if that is not an option, or if you are trying to implement some other form of complex keyboard interaction? In particular, as more desktop devices like Windows 8 and OS X add support for gestures, you can never be truly sure what capabilities a user has.

This blog posting was initially posted on my personal blog Unobfuscated.

Gestures To the Rescue

Gestures are your other option, and as long as you understand the limitations of JavaScript gestures as they relate to accessibility and AT interaction, you will be able to achieve some fantastic user interaction experiences.

The limitations of portable JavaScript Gestures

As of the date of this posting, JavaScript gesture libraries like hammer.js can reliably and portably generate the following list of gestures in HTML.

  • touch
  • release
  • hold
  • tap
  • doubletap
  • dragstart
  • drag
  • dragend
  • dragleft
  • dragright
  • dragup
  • dragdown
  • swipe
  • swipeleft
  • swiperight
  • swipeup
  • swipedown
  • transformstart
  • transform
  • transformend
  • rotate
  • pinch
  • pinchin
  • pinchout

The problem is that when the screen reader is turned on (e.g. VoiceOver on iOS), it intercepts the gestures and uses them to allow the VO user to more effectively navigate the applications on the device. So for example, the swipe right is used to move to the next element in the DOM and the swipe down is used to move to the next item in the current rotor settings. In a manner that is analogous to the keyboard event interception of a desktop screen reader, VoiceOver masks the events that are allowed through to JavaScript.

Double-Tap-Hold

Just as with desktop AT, there is a workaround, but it is not the same as with desktop AT (I wish it was a lot closer and I think there is a way for the mobile OSes to achieve a good user experience with more compatibility with ARIA, but that is for some future posting). The workaround on iOS is to double-tap and on the second tap, leave the finger on the surface for a second. This will make the VO cursor disappear and you can then perform the standard gesture (such as a swipe left, drag, hold etc.) The problem is that none of the gestures that require you to lift that finger, or use two fingers work. I have never been able to get a pinch to work in this scenario. As far as I can tell, the only gestures that reliably work in this mode are listed below (Note: I have removed the generic and the start and end events for brevity):

  • touch
  • release
  • hold
  • tap (use double-tap)
  • doubletap (use triple-tap)
  • dragleft
  • dragright
  • dragup
  • dragdown
  • swipeleft
  • swiperight
  • swipeup
  • swipedown

Which means that the following gestures cannot be relied-upon.

  • transform
  • rotate
  • pinchin
  • pinchout

Also, you will notice that there are no multi-finger-tap or -swipe gestures in the list.

Mapping gestures to keyboard events

The ARIA keyboard interaction for a date picker uses many different keyboard commands (in fact I chose it here because it is one of the most complex) that include the arrow and paging keys. Including:

  1. Up arrow – Move to the same day in the previous week
  2. Down arrow – move to the same day in the next week
  3. Left arrow – move to the previous day
  4. Right arrow – move to the next day
  5. Page-Up – Move to the same date in the previous month
  6. Page-Down – move to the same date in the next month
  7. Ctrl-Page-Up – move to the same date in the previous year
  8. Ctrl-Page-Down – move to the same date in the next year
  9. Home – move to the first of the month
  10. End – move to the last of the month
  11. ESC – dismiss the date picker
  12. Enter/Space – select a day and dismiss the date picker
  13. Tab – move focus out of the datepicker and onto the next focusable item
  14. Shift-Tab – move the focus out of the datepicker and to the previous focusable item

Touch, release and hold do not seem like events that map well to any of the above. So this leaves us with 10 events we can try to map to the 14 date picker events. Tab and shift tab can be reduced to the equivalent of an ESC and then a DOM navigation or rotor navigation (or just simply replace by a rotor navigation), which means we can probably live without those events. This means we have to eliminate just another two events. My candidates for this are the ones that move to the next and previous year. This is simply because in most scenarios, you are not choosing a date this far in advance or in the past and if you are, then typing the date is probably good enough for you. So for a standard generic mapping I suggest:

  1. Drag up (Up arrow) – Move to the same day in the previous week
  2. Drag down (Down arrow) – move to the same day in the next week
  3. Drag left (Left arrow) – move to the previous day
  4. Drag right (Right arrow) – move to the next day
  5. Swipe up (Page-Up) – Move to the same date in the previous month
  6. Swipe down (Page-Down) – move to the same date in the next month
  7. Swipe left (Home) – move to the first of the month
  8. Swipe right (End) – move to the last of the month
  9. Double-tap (ESC) – dismiss the date picker
  10. Tap (Enter/Space) – select a day and dismiss the date picker

This mapping leads to a nice interaction where you can double-tap-hold and then drag up, down, left and right to find a date in the current month – even moving across month boundaries and then easily also get to next and previous month. Performing similar mappings for other widgets can give you similarly pleasing results.

Getting hammer.js to reliably detect the swipes

If you’ve played with hammer.js as a gesture library, you may have noticed that the swipe gestures do not work with VoiceOver turned on. The reason for this is due to two default settings that get in the way:

  1. The swipe gestures all require one tap gesture at the beginning of the sequence, but because of iOS’ workaround requiring that the finger rest on the surface for 1s and then not be raised again, this tap never occurs. Changing hammer.js to allow 0 taps at the beginning of a swipe gesture will solve this problem
  2. The minimum velocity for a swipe gesture to be recognized is 0.7. I found that on an iPhone, I would have difficulty reliably getting my swipe speed above that. This can lead to frustration. A value of 0.5 allows for much more reliable swipe pickup. You may have to play with a value between 0.7 and 0.5 depending on your application’s needs and the affect this has on drag event detection.

Be one of the first to read new postings – follow me on Twitter @dylanbarrell

[hs_action id=”9813″]

Leave a Reply

You can use these HTML tags:

<a href="" title="" rel=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>