Leaking DRI fds when vaInitialize() is called again on same VADisplay after failure

[cgutman]

vaInitialize() doesn't clean up properly on failure, so subsequent calls to vaInitialize() will lead to leaks of a /dev/dri/card or /dev/dri/renderD file descriptor. Eventually the fd limit is reached and no more files can be opened, leading to a crash with a message similar to:

XIO:  fatal IO error 24 (Too many open files) on X server ":0"
      after 6 requests (6 known processed) with 0 events remaining.

Attempting to call vaInitialize() again after a failure might seem contrived at first glance, but it is a reasonable thing to do for clients that might be using vaSetDriverName() to try to get one driver over another.

Either the documentation should clearly state that vaTerminate() must be called even after a failed vaInitialize() call (which invalidates the whole VADisplay), or a failed vaInitialize() should leave the VADisplay in a clean state to allow another initialization attempt.

Reproducer:

gcc valeak.c -o valeak -lva -lva-x11 -lX11

#include <va/va.h>
#include <va/va_x11.h>
#include <X11/Xlib.h>

#include <stdlib.h>
#include <stdio.h>

int main() {
    Display *x11_disp = XOpenDisplay(NULL);
    if (!x11_disp) {
        fprintf(stderr, "Unable to open X11 display\n");
        return -1;
    }

    VADisplay va_disp = vaGetDisplay(x11_disp);
    if (!va_disp) {
        fprintf(stderr, "Unable to open VA display\n");
        return -1;
    }

    for (;;) {
        int major, minor;
        vaSetDriverName(va_disp, "NotARealDriver");
        VAStatus status = vaInitialize(va_disp, &major, &minor);
        if (status == VA_STATUS_SUCCESS) {
            fprintf(stderr, "vaInitialize() succeded with bogus driver name?\n");
            abort();
        }
        else {
            fprintf(stderr, "vaInitialize() failed: %s\n", vaErrorStr(status));
        }
    }
    
    vaTerminate(va_disp);
    XCloseDisplay(x11_disp);
}

Running lsof on the reproducer will show hundreds of leaked /dev/dri fds:

valeak  30432 cgutman  548u   CHR            226,128      0t0     571 /dev/dri/renderD128
valeak  30432 cgutman  549u   CHR            226,128      0t0     571 /dev/dri/renderD128
valeak  30432 cgutman  550u   CHR            226,128      0t0     571 /dev/dri/renderD128
valeak  30432 cgutman  551u   CHR            226,128      0t0     571 /dev/dri/renderD128
valeak  30432 cgutman  552u   CHR            226,128      0t0     571 /dev/dri/renderD128
valeak  30432 cgutman  553u   CHR            226,128      0t0     571 /dev/dri/renderD128
valeak  30432 cgutman  554u   CHR            226,128      0t0     571 /dev/dri/renderD128
valeak  30432 cgutman  555u   CHR            226,128      0t0     571 /dev/dri/renderD128
valeak  30432 cgutman  556u   CHR            226,128      0t0     571 /dev/dri/renderD128
valeak  30432 cgutman  557u   CHR            226,128      0t0     571 /dev/dri/renderD128
valeak  30432 cgutman  558u   CHR            226,128      0t0     571 /dev/dri/renderD128
valeak  30432 cgutman  559u   CHR            226,128      0t0     571 /dev/dri/renderD128
valeak  30432 cgutman  560u   CHR            226,128      0t0     571 /dev/dri/renderD128
valeak  30432 cgutman  561u   CHR            226,128      0t0     571 /dev/dri/renderD128
valeak  30432 cgutman  562u   CHR            226,128      0t0     571 /dev/dri/renderD128
valeak  30432 cgutman  563u   CHR            226,128      0t0     571 /dev/dri/renderD128
valeak  30432 cgutman  564u   CHR            226,128      0t0     571 /dev/dri/renderD128
valeak  30432 cgutman  565u   CHR            226,128      0t0     571 /dev/dri/renderD128
valeak  30432 cgutman  566u   CHR            226,128      0t0     571 /dev/dri/renderD128
valeak  30432 cgutman  567u   CHR            226,128      0t0     571 /dev/dri/renderD128
valeak  30432 cgutman  568u   CHR            226,128      0t0     571 /dev/dri/renderD128
valeak  30432 cgutman  569u   CHR            226,128      0t0     571 /dev/dri/renderD128
valeak  30432 cgutman  570u   CHR            226,128      0t0     571 /dev/dri/renderD128
valeak  30432 cgutman  571u   CHR            226,128      0t0     571 /dev/dri/renderD128
valeak  30432 cgutman  572u   CHR            226,128      0t0     571 /dev/dri/renderD128
valeak  30432 cgutman  573u   CHR            226,128      0t0     571 /dev/dri/renderD128
valeak  30432 cgutman  574u   CHR            226,128      0t0     571 /dev/dri/renderD128
valeak  30432 cgutman  575u   CHR            226,128      0t0     571 /dev/dri/renderD128

[XinfengZhang]

there are two cases:

1. DRM: the fd is opened by application, so libva could not close it. it is application responsibility to close the fd
2. X11,Wayland: the fd is opened by libva, libva should destroy it by calling pDisplayContext->vaDestroy

[XinfengZhang]

please help to review #753