Video Notes

Introduction

This document describes my setup for taking notes from videos, using Emacs and org-mode.

My goals:

Embed & playback video links into my notes
Generate, embed & post process animated from my notes

Use cases

BJJ

A very common use case for me is note taking from BJJ instructionals. Such videos are long in duration and describe sequences that you need to iterate at least a couple of times in order to fully understand. Often it helps converting such sequences into:

UML diagrams
Flashcards
Animated images or short clips

So, I needed a way to link a particular note to a particular point in the video and also capture parts of such videos as animated images so that I can embed in my notes.

Conference talk note taking

I am often watching conference talks on my computer. Usually, those talks are downloaded locally (to make sure, I can find them again if needed). Often, I need to place a bookmark at a particular point in the talk, or even create a link (as a reference) in one of my talks, blogs etc. Since, all these are handled by org-mode, it was important to be able to link org entries to points in media files.

Solution

I am already using bongo as media player linked for things like elfeed, so it made sense to base my solution to that. If you want to learn more about the combination above do watch: Emacs podcast manager with bongo+elfeed.

This literate blog contains my whole setup and should be an autonomous unit that you can drop in to you configuration using org-babel.

How to use this file:

I have this file under: `~/Documents/org/roam/video-notes.org`. The file is used both as a literate config file and also as the place where my notes are being stored.

I load this file using:

(org-babel-load-file "~/Documents/org/roam/video-notes.org")

Each time I playback videos using bongo, an instancee of `mplayer` is started behind the scenes, that communicates with Emacs in order to get information like the file being currently playing and the elapsed time.

From `org-capture` (usually bound to `C-c C-c`) I press `v n` to create a note. An org entry pointing to the particular point in the video will be created and stored in this file under `Notes`. Each time I need to visit that particular video, I just need to visit that particular entry and invoke the command `iocanel/bongo-play-org-entry-at-point`.

To capture parts of the video playing as an animation I need to mark the start and the end of the animation using `iocanel/bongo-mark-selection-start` (C-c b m s) and `iocanel/mark-selection-end` (C-c b m e) respectively. From `org-capture` (usually bound to `C-c C-c`) I press `v a` to create a note with an embeded animation. To fine tune the animation, you can modify the animation parameters, which are added as properties to entry and then invoke `iocanel/recreate-animation-at-point`.

The attached image, can be displayed inline using: `org-display-inline-images`. To watch the animation from within emacs you can follow this guide: Displaying animated gifs from within Emacs

Below you can find:

templates
functions
configuration
Place holder for the notes

Templates

The template below describes the structure of the enry created when I want to capture a point in a particular media file. It pretty much helps me create an entry with properties like:

timestamp

* %^{Video Name} :video:
:PROPERTIES:
:URL: %(iocanel/bongo-currently-playing-url)
:TIMESTAMP: %(iocanel/bongo-currently-playing-elapsed-time) 
:END:
%?

* %(iocanel/bongo-choose-animation-name) :animation:
:PROPERTIES:
:URL: %(iocanel/bongo-choose-selection-url)
:ANIMATION_START: %(iocanel/bongo-choose-selection-start)
:ANIMATION_END: %(iocanel/bongo-choose-selection-end)
:END:
[[%(iocanel/bongo-animation-output-file)]]
%(iocanel/bongo-create-animation)
%?

Functions

This section provides some helper functions for:

media -> org entry
org -> media

If not obvious, this code is used to jump from media to org entries and vice versa.

(require 'bongo)

;;
;; State
;; 
(defvar iocanel/bongo-animation-name nil "Animation name")
(defvar iocanel/bongo-selection-url nil "Selection url")
(defvar iocanel/bongo-selection-start nil "Selection start")
(defvar iocanel/bongo-selection-end nil "Selection end")

;;
;; Utilities
;; 
(defun iocanel/strip-file-scheme (url)
  "Mark the start of the selection."
  (interactive)
  (if (and url (s-starts-with? "file://" url))
      (substring url 7)
    url))

(defun iocanel/bongo-choose-animation-name ()
  "Choose selection url. This function is meant to be used from within templates so that selected value is used to update the selection url."
  (setq iocanel/bongo-animation-name (completing-read "Animation name:" `(,(iocanel/strip-file-scheme
                                                                            (file-name-sans-extension (file-name-nondirectory (iocanel/bongo-currently-playing-url))))) (lambda (a) t) nil)))

(defun iocanel/bongo-choose-selection-url ()
  "Choose selection url. This function is meant to be used from within templates so that selected value is used to update the selection url."
  (setq iocanel/bongo-selection-url (completing-read "Selection url:" `(,(iocanel/bongo-currently-playing-url)) (lambda (a) t) nil)))


;; Selection start
(defun iocanel/bongo-choose-selection-start ()
  "Choose selection start. This function is meant to be used from within templates so that selected value is used to update the selection start."
  (setq iocanel/bongo-selection-start (completing-read "Selection start:" `(,(iocanel/bongo-get-selection-start)) (lambda (a) t) nil)))

(defun iocanel/bongo-get-selection-start ()
 "Return the selection start"
 (cond ((stringp iocanel/bongo-selection-start) iocanel/bongo-selection-start)
       ((numberp iocanel/bongo-selection-start) (number-to-string iocanel/bongo-selection-start))
       (t nil)))

(defun iocanel/bongo-mark-selection-start ()
  "Mark the start of the selection."
  (interactive)
  (setq iocanel/bongo-selection-start (iocanel/bongo-currently-playing-elapsed-time)))

;; Selection end
(defun iocanel/bongo-choose-selection-end ()
  "Choose selection end. This function is meant to be used from within templates so that selected value is used to update the selection end."
  (setq iocanel/bongo-selection-end (completing-read "Selection end:" `(,(iocanel/bongo-get-selection-end)) (lambda (a) t) nil)))

(defun iocanel/bongo-get-selection-end ()
 "Return the selection end"
 (cond ((stringp iocanel/bongo-selection-end) iocanel/bongo-selection-end)
       ((numberp iocanel/bongo-selection-end) (number-to-string iocanel/bongo-selection-end))
       (t nil)))

(defun iocanel/bongo-mark-selection-end ()
  "Mark the end of the selection."
  (interactive)
  (setq iocanel/bongo-selection-end (iocanel/bongo-currently-playing-elapsed-time)))


;; Animation
(defun iocanel/bongo-create-animation ()
  "Create an animation from the selection start to the selection end."
  (interactive)
  (let* ((url (or iocanel/bongo-selection-url (iocanel/bongo-currently-playing-url)))
         (path (iocanel/strip-file-scheme url))
         (dir (file-name-directory path))
         (name (or iocanel/bongo-animation-name (file-name-sans-extension (file-name-nondirectory path))))
         (output-file (concat dir name ".gif"))
         (start (bongo-format-seconds (string-to-number (iocanel/bongo-get-selection-start))))
         (end (bongo-format-seconds (string-to-number (iocanel/bongo-get-selection-end)))))
    (iocanel/bongo-do-create-animation url start end name)))


(defun iocanel/bongo-do-create-animation (url start end &optional name framerate scale)
  "Create an animation using the URL from the selection START to the selection END. Optionally, specify the output NAME the FRAMERATE and SCALE."
  (interactive)
  (let* ((path (iocanel/strip-file-scheme url))
         (dir (file-name-directory path))
         (name (or name iocanel/bongo-animation-name (file-name-sans-extension (file-name-nondirectory path))))
         (output-file (concat dir name ".gif"))
         (framerate (or framerate 5))
         (scale (or scale "512:-1")))
    (if (and path start end)
        (progn
          (message "Creating animated gif: %s for video: %s in dir:%s" output-file path dir)
          (async-shell-command (format "ffmpeg -y -i %s -r %s -vf scale=%s -ss %s -to %s %s"
                                       (shell-quote-argument path)
                                       framerate
                                       scale
                                       start
                                       end
                                       (shell-quote-argument output-file))))
      output-file
      (progn
        (message "Can't create animated gif for file: %s bounds (%s - %s)." path start end)
        nil))))


(defun iocanel/bongo-animation-output-file ()
  "Returns the path to the animation output file."
  (let* ((url (or iocanel/bongo-selection-url (iocanel/bongo-currently-playing-url)))
         (path (iocanel/strip-file-scheme url))
         (dir (file-name-directory path))
         (name (or iocanel/bongo-animation-name (file-name-sans-extension (file-name-nondirectory path))))
         (output-file (concat dir name ".gif")))
    output-file))

;;
;; Extract information from bongo player
;;
(defun iocanel/bongo-currently-playing-elapsed-time()
  "Return the elapsed time of the media file currently playing."
  (interactive)
  (bongo-elapsed-time))

(defun iocanel/bongo-currently-playing-url ()
  "Return the url of the media file currently playing."
  (interactive)
  (with-bongo-playlist-buffer
    (cdr (assoc 'file-name (cdr bongo-player)))))

;;
;; Org entries
;;
(defun iocanel/bongo-play-org-entry-at-point ()
  "Play the play the media file linked to the org-entry at point."
  (interactive)
  (let* ((p (point))
         (url  (org-entry-get nil "URL"))
         (elapsed (org-entry-get nil "ELAPSED")))
    (with-bongo-playlist-buffer
      (bongo-insert-uri url)
      (bongo-previous-object)
      (bongo-play)
      (when (stringp elapsed) (bongo-seek-to (string-to-number elapsed))))))

(defun iocanel/bongo-recreate-animation-at-point ()
  "Recreate  the animation that corresponds to the particular element. This allows tuning of parameters"
  (interactive)
  (let* ((p (point))
         (url  (org-entry-get nil "URL"))
         (name  (org-get-heading t t t t))
         (framerate  (org-entry-get nil "FRAMERATE"))
         (scale  (org-entry-get nil "SCALE"))
         (start  (org-entry-get nil "ANIMATION_START"))
         (end  (org-entry-get nil "ANIMATION_END")))
    (message "Recreating animation for: %s using file: %s (%s-%s)" name url start end)
    (iocanel/bongo-do-create-animation url start end name framerate scale)))

;;
;; Bindings
;;
(global-set-key (kbd "C-c b m s") 'iocanel/bongo-mark-selection-start)
(global-set-key (kbd "C-c b m e") 'iocanel/bongo-mark-selection-end)
(global-set-key (kbd "C-c b g") 'iocanel/bongo-create-animation)


;;
;; Advices
;;                                        

(defadvice bongo-play-line (after bongo-play-line-around activate)
  "Check if bluetooth is connected and use pulse audio driver."
  (interactive)
  (setq iocanel/bongo-selection-url (iocanel/bongo-currently-playing-url)))

Configuration

For this setup to we work we need to configure org-catpure. I want all my video notes to land under `~/Documents/org/roam/video-notes.org` and from there I can move notes as I feel like.

(setq org-capture-templates (append org-capture-templates '(
                                                            ("v" "Video")
                                                            ("vn" "Notes"  entry (file+olp "~/Documents/org/roam/video-notes.org" "Notes")(file "~/Documents/org/templates/video-note.orgtmpl"))
                                                            ("vg" "Animation"  entry (file+olp "~/Documents/org/roam/video-notes.org" "Notes")(file "~/Documents/org/templates/animation.orgtmpl")))))

Notes

This is the place where captured notes are being added. From here notes can be refiled to where they belong.

The entries below are just examples:

Straight ankle lock

This is an animated gif of how Mickey Musumeci performs the straight ankle lock. Original video can be found at: https://www.youtube.com/watch?v=JD3QucUWIwI

iocanel/video-notes.org