Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Capture moving plot to Video file #4387

Open
IanMayo opened this issue Jul 11, 2019 · 9 comments · May be fixed by #5167
Open

Capture moving plot to Video file #4387

IanMayo opened this issue Jul 11, 2019 · 9 comments · May be fixed by #5167
Assignees
Labels

Comments

@IanMayo
Copy link
Member

IanMayo commented Jul 11, 2019

An analyst is struggling to get high quality video recordings of Debrief. His IT support organisation can only supply VLC, and it's proving a real challenge to get tidy video out of that. It's only works intermittently, and then only with full screen capture.

The work-flow would be streamlined if this was possible:

  • choose whether to record just the plot, or the whole application (and this allow time graphs to be captured, too)
  • choose whether to record to video or animated GIF
  • choose destination (maybe this can be done on completion)
  • when storing video, try to store keyframes + deltas, to reduce video file size

Hopefully we can integrate it into the new Record button in the Debrief time controller.

Here's a possible library:
http://www.randelshofer.ch/monte/
and a demo of using it here:
https://unmesh.me/2012/01/13/recording-screencast-of-selenium-tests-in-java/

Alternatively, it may be possible to use Xuggler to capture the video:
https://gist.github.com/e-d/5f116cb72f0667467f33

(we already integrate Xuggler)

@clenoble
Copy link

clenoble commented Oct 1, 2019

@IanMayo Rather than use the record button in the time controller, I would move both export options (ppt and video) to the existing Export menu under File. This would allow you to have a Wizard with a few steps to enter the required parameters. There would be no need for the user to press "record", wait for the animation to run and then "stop" at the exact right time. The Wizard could let them choose between exporting the entire duration, only the duration selected in the time controller (and possibly a 3rd option with free input time stamp for start and end).
If generating the file takes time, there can be a loading animation with a countdown until the file is ready to download.

@IanMayo
Copy link
Member Author

IanMayo commented Oct 9, 2019

@clenoble - I think the user needs more involvement in the export. The user will control:

  • the viewport (area of the map that's exported). The viewport could also change if the user wants to pan/zoom during the recording
  • which tracks/shapes are visible
  • the time the export start
  • the length of the export
  • how frequent the screen is redrawn during the replay (step interval from 1 step/sec to 10 steps/sec)
  • how far forward in scenario time for each step (1 sec, 1 min, 5 min...)

Some of these we could infer, such as the area of coverage (use the currently visible area) or the time of coverage (from the time slider bars). But, the other are sensitive to the data that's being looked at - which influences how the message is conveyed.

(I recognise that it's quite common to load a video and export it to a different video, but that just involves some resampling and/or storing to a different type. But, this export seems more involved).

@clenoble
Copy link

@IanMayo Most items can be selected from a wizard, but the important point you make is that a user might want to interact with the views while recording (showing / hiding elements, zooming in / out). In this case, the record button is required. I would even add a pause / resume button, rather than just stop.
Another possibly useful feature might be to let user record a verbal comment at the same time as recording the video? Is that something that our users would be interested in?

For the UI, I suggest using controls similar to screen-o-matic https://screencast-o-matic.com. See screenshot below:
screen-o-matic-UI
Of course, we would adapt the settings panel to what we need.

This would allow the user to switch from one XY plot to another during the video, but not record the time controller and outline if they think it's not useful.

video-recording

@IanMayo
Copy link
Member Author

IanMayo commented Oct 11, 2019

Thanks @clenoble .

(I hope we can avoid re-writing screencast-o-matic)

You have raised something I hadn't considered. My original idea was to let the user select between recording just the plot [1], or recording the whole application [2].

Option [2] would allow one or more graphs to be included, or maybe the Narrative Viewer if the "diary" of what happened was relevant. Other views (such as the Outline & Tote) would be collapsed.

But, in order to "drive" the recording, the analyst needs the Time Controller to be open. It's unlikely they would want the whole of the left-hand column. But, the time-controller could be dragged down, to occupy the left-hand "end" of the graph:

image

Aah, that's still sub-optimal sine there's the wasted space for the toolbar & menu.

Hmm, ideally we'd avoid the step to drag to select the target area. Aah, hold on. We know the dimensions of the menu, toolbar, status bar. We could calculate the viewport required that would exclude them.

Hmm (2). But, the user may want the status bar included, if the current mouse position is important.

@clenoble
Copy link

@IanMayo I don't think dragging to select the area is such a hurdle for the user, on the contrary, it's much more intuitive and flexible than selecting specific views in a wizard, or re-organizing the entire UI. It might be more complex from a technical point of view.

Another idea would be to add two icons in the toolbar, one for PPT, one for video. Clicking on either of them would enable the "Start record" button (otherwise disabled), and open a setting panel. For PPT, the user can set the template. For video, they can set the views, re-draw frequency and time to step forward. For the visibility of tracks, I think it would be best if tracks visible on screen will be visible on the video. The user can hide tracks in the outline panel if they want to hide them in the video. Otherwise the user will be more likely to make errors if what he sees on screen doesn't match what's recorded.
time-controller-ppt-video-setting

They will manage the start and end of recording using the start record / stop record button.

@IanMayo
Copy link
Member Author

IanMayo commented Jul 24, 2020

We've suspended this task, I think we were struggling with xuggler. If/when we restart it, we should consider one of the ffmpeg java wrapper libraries - since ffmpeg is still actively maintained.

@IanMayo
Copy link
Member Author

IanMayo commented Jan 13, 2021

An alternate strategy could be to use a Python record-to-video utility, as created by Ed.

from threading import Thread, Event
import time

import PIL
import PIL.ImageGrab
import PIL.ImageChops
import PIL.Image

import moviepy.editor as mpy





# image quality (1-95) for gifs
qual = 95
# time between frame capture attempt (s)
sleepTime = 0.4

#left, bottom, right, top of the area to capture
#0,0 is to left corner of screen, down is +ve

x0 = 0 
y0 = 50 
x1 = 1800
y1 = 1140

# how many frames of animation to take
maxFrames = 10

#used for speed of gifs / movies
frames_second = 4

# width of the gif output (height scales to this)
outWidth = 640

# prefix to filenames
image_title = "animation_" 
# working folder
save_folder = 'C:\\My data\\image_grab\\'





count = 0
event = Event()

imArray = []
frameList = []



    
def screen_cap(frameCount,imList,frames):
    uFrames = 0
    while True:
        
        im = PIL.ImageGrab.grab(bbox =(x0, y0, x1, y1))  


        if event.is_set():
            break 
        
        if frameCount > 0:
            diff = PIL.ImageChops.difference(im, old_im)

            if diff.getbbox():
                uFrames = uFrames + 1
                print("Frame " + str(frameCount) + " NEW - frame #:" + str(uFrames) + " / " + str(maxFrames))
                im.save(save_folder + image_title + str(uFrames).zfill(4) + '.jpg', quality=qual)
                    
                imOut = im
                wPercent = (outWidth / float(imOut.size[0]))
                hSize = int((float(imOut.size[1])*float(wPercent)))
                imOut = imOut.resize((outWidth,hSize),PIL.Image.ANTIALIAS)
                
                frames.append(save_folder + image_title + str(uFrames).zfill(4) + '.jpg')
                imList.append(imOut)
                
            else:
                print("Frame " + str(frameCount) + " DUPLICATE ")
        else:
            print("loading first frame")
           
            
        
        old_im = im
        frameCount = frameCount + 1
        time.sleep(sleepTime)

    print('Finising capture - please wait')


my_count = 0
t = Thread(target=screen_cap, args=(my_count, imArray,frameList ))
t.start()


print("ctrl-C to exit!")
        
while True:
    try:
        if len(frameList) > maxFrames - 1:
            print("Reached max frames")
            event.set()
            break
        #print(my_count)
        time.sleep(0.1)
    except KeyboardInterrupt:
        event.set()
        print("Stopping capture")
        break


t.join()

print("generating gifs")

gifTitle = image_title + '_loop.gif'
mp4Title = image_title + '.mp4'

imArray[0].save(gifTitle,save_all=True, append_images=imArray[1:], optimize=True, duration= 1000 / frames_second, loop=1)

gifTitle = image_title + '_no_loop.gif'

imArray[0].save(gifTitle,save_all=True, append_images=imArray[1:], optimize=True, duration= 1000 / frames_second, loop=0)

print("generating mp4")
clip = mpy.ImageSequenceClip(frameList, fps=frames_second)
clip.write_videofile(mp4Title)

print("done" )

@IanMayo
Copy link
Member Author

IanMayo commented Feb 19, 2021

An alternative route to this could be to use an existing application.

We may be able to modify this Open Source application to include features we would need to integrate it into Debrief:
https://www.cockos.com/licecap/

@IanMayo
Copy link
Member Author

IanMayo commented Apr 19, 2021

Here's a tutorial on how to use JCodec to make a screen recorder:
https://steemit.com/utopian-io/@ideba/how-to-build-a-program-for-pc-screen-recording-video-with-java-part-2

Sources for above here: https://github.com/enyason/DesktopScreenRecorder

Here is an Eclipse plugin that claims to provide screen recording:
https://marketplace.eclipse.org/content/eclipse-screen-video-capturing-plugin

@IanMayo IanMayo linked a pull request Feb 24, 2023 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants