SDL 2 full screen badly broken in Linux with Nvidia drivers
I'm running Ubuntu 16.10 with Xfce and proprietary Nvidia drivers, version 340.101-0ubuntu0.16.10.1. The graphics hardware is GeForce 8600 GT. Although this is not the latest Nvidia driver, it is the latest which supports this card. I see the same behaviour with both with SDL 2.0.5 downloaded from this site and built myself, and Ubuntu's libsdl2-2.0-0:amd64 2.0.4+dfsg2-1ubuntu1.
When I switch a window to full screen using SDL_SetWindowFullscreen(window, SDL_WINDOW_FULLSCREEN), the size reported to the application is larger than the video mode. Initially I see the top left part of the screen. I can scroll around the application output by bringing my mouse to the edge of the screen. For example going to the right edge scrolls the viewport to the right, and sees what was hidden there, while scrolling stuff at the left edge out of view. So, I can see all of the application by scrolling around, and the scrolling is limited to application output size, but this is not a nice way to use an application.
Afterwards, when I switch back to windowed mode using SDL_SetWindowFullscreen(window, 0), the original desktop video mode is restored if and only if the viewport was scrolled all the way up and all the way left when I switched back. Otherwise, I stay in the same resolution used by the full screen mode, but see the desktop, and need to scroll around with the mouse to view it. Recovery is possible using xrandr at the command line, or by going back into full screen mode, and back into windowed mode, making sure the viewport was at the top left.
This problem only occurs with SDL 2 and does not occur with SDL 1 libsdl1.2debian:amd64 1.2.15+dfsg1-4 SDL_WM_ToggleFullScreen().
SDL 2 by default uses Xrandr for mode switching. If I set SDL_VIDEO_X11_XRANDR=0, things get even worse. After switching to full screen mode, the screen is black, but I can scroll around with the mouse to view the rest of the desktop that's not covered by that blackness. The application is totally hung. Control-C does not work and I need to kill -9 it. SDL_VIDEO_X11_LEGACY_FULLSCREEN=1 messes up the same way. With both SDL_VIDEO_X11_XRANDR=0 and SDL_VIDEO_X11_XVIDMODE=0 there is no mode switching, and full screen is full screen at the desktop resolution. That works fine.
If I use the open source nouveau driver instead of Nvidia's proprietary driver, full screen works fine.