tvOS swipe left/right gesture support (patch enclosed)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

tvOS swipe left/right gesture support (patch enclosed)

oviano
I found that events weren't being generated for the simple directional swipes on the Apple TV touch remote.

While it's possible to use the touch press/release/motion events to detect simple swipes of this nature (and I had this working well enough) it struck me that this was re-inventing the wheel.

As such, I have written a patch that recognises the proper swipe up/down/left/right events issued in UIKit and passes these on as directional key presses. This also means that the touch remote operates the same way as the old-style IR remote in this respect.








Code:

diff -r 007dfe83abf8 src/video/uikit/SDL_uikitview.m
--- a/src/video/uikit/SDL_uikitview.m   Wed Oct 19 20:50:33 2016 -0700
+++ b/src/video/uikit/SDL_uikitview.m   Sun Dec 18 14:57:42 2016 +0300
@@ -48,7 +48,20 @@
 #if !TARGET_OS_TV
         self.multipleTouchEnabled = YES;
 #endif
-
+#if TARGET_OS_TV
+        UISwipeGestureRecognizer *swipeUp = [[UISwipeGestureRecognizer alloc] initWithTarget:self action:@selector(swipeGesture:)];
+        swipeUp.direction = UISwipeGestureRecognizerDirectionUp;
+        [self addGestureRecognizer:swipeUp];
+        UISwipeGestureRecognizer *swipeDown = [[UISwipeGestureRecognizer alloc] initWithTarget:self action:@selector(swipeGesture:)];
+        swipeDown.direction = UISwipeGestureRecognizerDirectionDown;
+        [self addGestureRecognizer:swipeDown];
+        UISwipeGestureRecognizer *swipeLeft = [[UISwipeGestureRecognizer alloc] initWithTarget:self action:@selector(swipeGesture:)];
+        swipeLeft.direction = UISwipeGestureRecognizerDirectionLeft;
+        [self addGestureRecognizer:swipeLeft];
+        UISwipeGestureRecognizer *swipeRight = [[UISwipeGestureRecognizer alloc] initWithTarget:self action:@selector(swipeGesture:)];
+        swipeRight.direction = UISwipeGestureRecognizerDirectionRight;
+        [self addGestureRecognizer:swipeRight];
+#endif
         touchId = 1;
         SDL_AddTouch(touchId, "");
     }
@@ -56,6 +69,27 @@
     return self;
 }
 
+#if TARGET_OS_TV
+- (void)swipeGesture:(UISwipeGestureRecognizer *)sender
+{
+    switch (sender.direction)
+    {
+    case UISwipeGestureRecognizerDirectionUp:
+        SDL_SendKeyboardKey(SDL_PRESSED, SDL_SCANCODE_UP);
+        break;
+    case UISwipeGestureRecognizerDirectionDown:
+        SDL_SendKeyboardKey(SDL_PRESSED, SDL_SCANCODE_DOWN);
+        break;
+    case UISwipeGestureRecognizerDirectionLeft:
+        SDL_SendKeyboardKey(SDL_PRESSED, SDL_SCANCODE_LEFT);
+        break;
+    case UISwipeGestureRecognizerDirectionRight:
+        SDL_SendKeyboardKey(SDL_PRESSED, SDL_SCANCODE_RIGHT);
+        break;
+    }
+}
+#endif
+
 - (void)setSDLWindow:(SDL_Window *)window
 {
     SDL_WindowData *data = nil;

_______________________________________________
SDL mailing list
[hidden email]
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org
Reply | Threaded
Open this post in threaded view
|

Re: tvOS swipe left/right gesture support (patch enclosed)

Alex Szpakowski
SDL is already supposed to send left/right/up/down key presses when “press” events are detected on the Apple TV Remote ( https://developer.apple.com/library/content/documentation/General/Conceptual/AppleTV_PG/DetectingButtonPressesandGestures.html ) – does that work for you? Does your extra gesture recognizer code interfere with that at all?

On Dec 18, 2016, at 8:15 AM, oviano <[hidden email]> wrote:

I found that events weren't being generated for the simple directional swipes on the Apple TV touch remote.

While it's possible to use the touch press/release/motion events to detect simple swipes of this nature (and I had this working well enough) it struck me that this was re-inventing the wheel.

As such, I have written a patch that recognises the proper swipe up/down/left/right events issued in UIKit and passes these on as directional key presses. This also means that the touch remote operates the same way as the old-style IR remote in this respect.








Code:

diff -r 007dfe83abf8 src/video/uikit/SDL_uikitview.m
--- a/src/video/uikit/SDL_uikitview.m   Wed Oct 19 20:50:33 2016 -0700
+++ b/src/video/uikit/SDL_uikitview.m   Sun Dec 18 14:57:42 2016 +0300
@@ -48,7 +48,20 @@
 #if !TARGET_OS_TV
         self.multipleTouchEnabled = YES;
 #endif
-
+#if TARGET_OS_TV
+        UISwipeGestureRecognizer *swipeUp = [[UISwipeGestureRecognizer alloc] initWithTarget:self action:@selector(swipeGesture:)];
+        swipeUp.direction = UISwipeGestureRecognizerDirectionUp;
+        [self addGestureRecognizer:swipeUp];
+        UISwipeGestureRecognizer *swipeDown = [[UISwipeGestureRecognizer alloc] initWithTarget:self action:@selector(swipeGesture:)];
+        swipeDown.direction = UISwipeGestureRecognizerDirectionDown;
+        [self addGestureRecognizer:swipeDown];
+        UISwipeGestureRecognizer *swipeLeft = [[UISwipeGestureRecognizer alloc] initWithTarget:self action:@selector(swipeGesture:)];
+        swipeLeft.direction = UISwipeGestureRecognizerDirectionLeft;
+        [self addGestureRecognizer:swipeLeft];
+        UISwipeGestureRecognizer *swipeRight = [[UISwipeGestureRecognizer alloc] initWithTarget:self action:@selector(swipeGesture:)];
+        swipeRight.direction = UISwipeGestureRecognizerDirectionRight;
+        [self addGestureRecognizer:swipeRight];
+#endif
         touchId = 1;
         SDL_AddTouch(touchId, "");
     }
@@ -56,6 +69,27 @@
     return self;
 }
 
+#if TARGET_OS_TV
+- (void)swipeGesture:(UISwipeGestureRecognizer *)sender
+{
+    switch (sender.direction)
+    {
+    case UISwipeGestureRecognizerDirectionUp:
+        SDL_SendKeyboardKey(SDL_PRESSED, SDL_SCANCODE_UP);
+        break;
+    case UISwipeGestureRecognizerDirectionDown:
+        SDL_SendKeyboardKey(SDL_PRESSED, SDL_SCANCODE_DOWN);
+        break;
+    case UISwipeGestureRecognizerDirectionLeft:
+        SDL_SendKeyboardKey(SDL_PRESSED, SDL_SCANCODE_LEFT);
+        break;
+    case UISwipeGestureRecognizerDirectionRight:
+        SDL_SendKeyboardKey(SDL_PRESSED, SDL_SCANCODE_RIGHT);
+        break;
+    }
+}
+#endif
+
 - (void)setSDLWindow:(SDL_Window *)window
 {
     SDL_WindowData *data = nil;
_______________________________________________
SDL mailing list
[hidden email]
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org


_______________________________________________
SDL mailing list
[hidden email]
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org
Reply | Threaded
Open this post in threaded view
|

Re: tvOS swipe left/right gesture support (patch enclosed)

oviano
In reply to this post by oviano
That's great, but it doesn't work and no such events come through.

If you can point me in the direction of where the directional swipe gestures are handled in the code I can try and fix it, but I couldn't see it.

If you're talking about the old (non touch) remote then yes that generates the events correctly.

I'm talking about the swipes on the touch remote.

_______________________________________________
SDL mailing list
[hidden email]
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org
Reply | Threaded
Open this post in threaded view
|

Re: tvOS swipe left/right gesture support (patch enclosed)

Alex Szpakowski
https://hg.libsdl.org/SDL/file/f6cd81aab88e/src/video/uikit/SDL_uikitview.m#l204

On Dec 18, 2016, at 10:44 AM, oviano <[hidden email]> wrote:

That's great, but it doesn't work and no such events come through.

If you can point me in the direction of where the directional swipe gestures are handled in the code I can try and fix it, but I couldn't see it.

If you're talking about the old (non touch) remote then yes that generates the events correctly.

I'm talking about the swipes on the touch remote.
_______________________________________________
SDL mailing list
[hidden email]
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org


_______________________________________________
SDL mailing list
[hidden email]
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org
Reply | Threaded
Open this post in threaded view
|

Re: tvOS swipe left/right gesture support (patch enclosed)

Alex Szpakowski
I just tested on the tvOS Simulator and the key press events work for me with the latest SDL code from hg.

On Dec 18, 2016, at 10:45 AM, Alex Szpakowski <[hidden email]> wrote:

https://hg.libsdl.org/SDL/file/f6cd81aab88e/src/video/uikit/SDL_uikitview.m#l204

On Dec 18, 2016, at 10:44 AM, oviano <[hidden email]> wrote:

That's great, but it doesn't work and no such events come through.

If you can point me in the direction of where the directional swipe gestures are handled in the code I can try and fix it, but I couldn't see it.

If you're talking about the old (non touch) remote then yes that generates the events correctly.

I'm talking about the swipes on the touch remote.
_______________________________________________
SDL mailing list
[hidden email]
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org



_______________________________________________
SDL mailing list
[hidden email]
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org
Reply | Threaded
Open this post in threaded view
|

Re: tvOS swipe left/right gesture support (patch enclosed)

oviano
In reply to this post by oviano
Ok I'll try the simulator later but you're testing the swipe gestures, correct?

_______________________________________________
SDL mailing list
[hidden email]
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org
Reply | Threaded
Open this post in threaded view
|

Re: tvOS swipe left/right gesture support (patch enclosed)

Alex Szpakowski
The Apple TV Remote touchpad “tap” actions, as described here: https://developer.apple.com/tvos/human-interface-guidelines/remote-and-controllers/#buttons

The ‘tap’ actions map pretty cleanly to button presses, in terms of what their behaviour describes. SDL doesn’t really have analogous user-facing APIs that describe OS-handled swipe gestures in the same manner, however you can access the Remote’s touchpad as joystick axes in order to get a more granular description of what the user is doing with it.

On Dec 18, 2016, at 11:55 AM, oviano <[hidden email]> wrote:

Ok I'll try the simulator later but you're testing the swipe gestures, correct?
_______________________________________________
SDL mailing list
[hidden email]
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org


_______________________________________________
SDL mailing list
[hidden email]
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org
Reply | Threaded
Open this post in threaded view
|

Re: tvOS swipe left/right gesture support (patch enclosed)

oviano
In reply to this post by oviano
The up/down/left/right swipe gestures on Apple TVs are the primary method to navigate around button/menu systems on the device so assuming that some SDL apps will have button or menu systems that require the user to navigate from one button to another then it's far better to hook into these gestures...

Otherwise you are expecting the user to re-implement these themselves and inevitably there will be subtle differences in "feel" compared to the official ones leading to a substandard user experience.

_______________________________________________
SDL mailing list
[hidden email]
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org
Reply | Threaded
Open this post in threaded view
|

Re: tvOS swipe left/right gesture support (patch enclosed)

Alex Szpakowski
2 out of the 3 gesture types outlined here are implemented as key presses (click and tap) https://developer.apple.com/tvos/human-interface-guidelines/remote-and-controllers/#gestures 

Given the fact that key presses can’t represent swipe strength which is one of the main purposes of a swipe, and the fact that the direction key presses are already used by the tap gestures, how do you propose swipes should be exposed in SDL’s APIs?

On Dec 18, 2016, at 12:32 PM, oviano <[hidden email]> wrote:

The up/down/left/right swipe gestures on Apple TVs are the primary method to navigate around button/menu systems on the device so assuming that some SDL apps will have button or menu systems that require the user to navigate from one button to another then it's far better to hook into these gestures...

Otherwise you are expecting the user to re-implement these themselves and inevitably there will be subtle differences in "feel" compared to the official ones leading to a substandard user experience.
_______________________________________________
SDL mailing list
[hidden email]
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org


_______________________________________________
SDL mailing list
[hidden email]
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org
Reply | Threaded
Open this post in threaded view
|

Re: tvOS swipe left/right gesture support (patch enclosed)

Alex Szpakowski

> Given the fact that key presses can’t represent swipe strength which is one of the main purposes of a swipe

Actually I suppose the Apple documentation is a bit misleading about this. :)
_______________________________________________
SDL mailing list
[hidden email]
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org
Reply | Threaded
Open this post in threaded view
|

Re: tvOS swipe left/right gesture support (patch enclosed)

Alex Szpakowski
I pushed a commit which should do what you want: https://hg.libsdl.org/SDL/rev/a6babd973955

> On Dec 18, 2016, at 12:41 PM, Alex Szpakowski <[hidden email]> wrote:
>
>
>> Given the fact that key presses can’t represent swipe strength which is one of the main purposes of a swipe
>
> Actually I suppose the Apple documentation is a bit misleading about this. :)

_______________________________________________
SDL mailing list
[hidden email]
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org
Reply | Threaded
Open this post in threaded view
|

Re: tvOS swipe left/right gesture support (patch enclosed)

oviano
In reply to this post by oviano
Well the swipe gesture I'm talking about hooking into, and which my patch uses is just a simple standard swipe with direction only. There is concept of strength:

https://developer.apple.com/reference/uikit/uiswipegesturerecognizer

So it maps pretty easily to the arrow key presses.

It's funny - I actually had no idea (despite owning a touch-based Apple TV since they were available) that you could tap to produce the direction "buttons", I'd always swiped which is why it seems to me such a big omission. I simply had no way to navigate within the UI of my application (a video player) Smile

_______________________________________________
SDL mailing list
[hidden email]
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org
Reply | Threaded
Open this post in threaded view
|

Re: tvOS swipe left/right gesture support (patch enclosed)

oviano
In reply to this post by oviano
There is *no* concept of strength, I should say.

_______________________________________________
SDL mailing list
[hidden email]
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org
Reply | Threaded
Open this post in threaded view
|

Re: tvOS swipe left/right gesture support (patch enclosed)

oviano
In reply to this post by oviano
Thanks for making the change - I haven't tested your slight variation on my code yet but I will at some point and let you know if I encounter any problems.

_______________________________________________
SDL mailing list
[hidden email]
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org