Tame your music collection with Beets (orig blog post)

mybeetsconfig

I've finally tamed my large MP3 collection using the awesome open source utility called Beets.  That, along with a few other utilities and scripts, have allowed me easily clean up my existing library, import new music, sync with my Android phone, and then playback on my computer with an extremely simple but powerful query language.

Firstly, let's talk about the parameters for which I was seeking a solution:
Command line interface - Ideally, I should be able to use the solution on any OS platform without needing more than a Bash environment and some Unix utilities.  Also, with each cog in the machine being a Bash utility/script, it will make it easy to shim/pipe/redirect to other utilities to facilitate any other integrations I can dream up.
Minimal data outside of the MP3 files - I'd like as much of the MP3 data to be stored in the MP3 files themselves.  A small metadata database file is ok, if it only contains duplicate information (non-volatile) and is stored with the collection, for easy backing up.
Fix MP3 Tags - Need to be able to fix any missing/wrong MP3 tags in my existing collection, as well as new music that is imported.  Along those lines, I need to be able to configure custom rules to override some of what may come over as gospel from MusicBrainz, et al.
Playback via CLI MPlayer using queries - As is evident in item #1, I'm a command line guy, and so it should be no surprise that I've used MPlayer as my audio player for, like, ever and am accustomed to the commands.  In the past, I would either pipe an ad-hoc find or point to a pre-generated playlist M3U to listen to whatever genre or artist I was in the mood for.  Now, I was looking for a more natural query language to simplify things.
Android Sync - Need to be able to copy over the subset of my collection that I need on my phone, and then sync the delta of changes later on.
Backup to my home NAS - Although, I have my NAS already setup to backup my Mac via Time Machine, I like to keep my collection sync'd to a separate shared folder, in case I need to access it via a Linux box, or if my wife or daughter is after one of my tracks. 

Step #1 - Beets

So, given the above parameters, it didn't take long to come across Beets.  This fantastic piece of software solves most of my needs and is just limitless in its usage possibilities.  Once installed, my first order of business was to create a config file, in order to prepare for the initial import of my existing music collection.  I created a Beets configuration file at ~/.config/beets/config.yaml with the following minimal contents:

library:/music/library.db
directory:/music
  import:
    copy:yes
  paths:
    default:$genre/$albumartist/$album%aunique{}/$track-$title
    singleton:$genre/$artist/misc/$title
    comp:$genre/$albumartist/$album%aunique{}/$track-$title

This would create my collection in the /music folder (with the Beets database saved there as well).  It would also only do a copy on import (instead of a move) so I would have a chance to verify that everything was copacetic without dumping the original files.  I, then, tweaked the paths such that the top level folder is the genre, which is how I like to roll.

Time for importing.  I copied my existing collection over to /music_old and kicked off the import:

$ beet import /music_old

For each song/album it encounters, you'll get the suggested match from MusicBrainz, which you can accept, or you can attempt to edit tags on the fly.  I chose to initially take everything suggested, however, I did often hit "T" for "tracks", when I just had some misc files for a particular artist.  This made matching the single tracks easier, and causing them to end up in the "misc" folder under the artist's name, as specified in the config file under "singleton".

After completing this process manually (which took about an hour for 2500 or so songs), I noticed a number of artists that ended up in genres that I didn't really agree with.  On to step #2.

Step #2 - EyeD3

After having filled out all of the MP3 tags of my existing collection from MusicBrainz, I had a number of artists/albums/tracks that needed some genre fixing.  EyeD3 to the rescue.

The first order of business, after installation, was to check out the list of genre codes:

eyeD3 --plugin=genres

Armed with the correct genre id, I could update a given track:

eyeD3 -G 9 /music/Metal/Pantera/Cowboys\ From\ Hell/06-Domination.mp3

(of course, you can easily process multiple files, sprinkling in a little find and xargs, e.g.)

Any updates to MP3 tags requires you to let Beets know about it (more about queries in a bit), so it can update its database:

$ beet update Pantera

I was mostly moving whole artists over to other genres, so I quickly fashioned a more elaborate, single command:

$ ARTISTNAME=Jeremy\ Soule && find "$ARTISTNAME" -type f -print0 | xargs -0 eyeD3 -G 24 && beet update $ARTISTNAME


Step #3 - New Music Import

So, now my existing collection was looking pretty good.  Time to add in some new music I had purchased and downloaded from Amazon.

$ beet import ~/Music/Amazon\ MP3/Vanden\ Plas/Chronicles\ of\ the\ Immortals_\ Netherworld

Obviously, no real difference in this step vs. steps #1 and #2.

Step #4 - Play some music

Ok, after all that grunt work, we can enjoy the fruits of our labors - Beets' query language.

First thing to note is that the query syntax for Beets is used in many of its commands, such as ls and update.  The ls command is what we're after now, i.e. list the tracks that match my search criteria.  For example:

$ beet ls circus maximus  #find by artist
$ beet ls mercy falls  #find by album
$ beet ls through osiris eyes  #find by track
$ beet ls path:/music/Metal/Disturbed  #find my path

These examples will use each word as an individual keyword, so quote the combination of search terms as needed.  Also, additional prefix qualifiers exist, like "artist:" and "album:", for such cases as an artist having the same name as another song in your collection.  There's much more to the query language, so take a look here: Beets Query Reference

Great!  That gives the ability to create a targeted playlist on the fly.  Now, let's get those results over to MPlayer for playback:

$ beet ls -p genre:Metal | mplayer -shuffle -playlist /dev/fd/3 3<&0 <

Bam!  Now randomly playing through all songs in the Metal genre.  Pretty good, but man that command is pretty long.  One last task to make the whole solution shine - create a music.sh shell script to run the mplayer command:

#! /bin/bash
OIFS="$IFS"
IFS=$'\n'
beet ls -p $@ | mplayer -shuffle -playlist /dev/fd/3 3<&0 </dev/tty
IFS="$OIFS"

and then to add an alias to my ~/.profile:

alias play='/Users/jasongardner/scripts/music.sh $@'

Aaaahhhh.  Now we're talking!  Now playing anything from my collection is dead simple:

$ play Iron Maiden  #artist
$ play Guns Roses Appetite Destruction  #album
$ play Tool Sober  #track
# play Savatage, but not the songs I have under the Christmas genre
$ play path:/music/Progressive\ Rock/Savatage


Step #5

Everything being in place, it's time to backup to the shared folder on my NAS.

$ rsync -avz --delete /music/ [email protected]:/volume1/shared/jason/music/

Pretty straightforward, one-way sync to a remote share.

Step #6

Lastly, I wanted get a particular subset of my collection onto my phone.  I use SSHelper (an SSH server app) on my phone (currently have a HTC One M8), so rsync was the logical choice to perform the sync.  After some trial and error, I settled on this approach.  First, I wrote a small script to spit out a text file containing the paths that I wanted to sync:

#!/bin/bash
echo "Creating Playlist for phone sync..."
echo -n > "/music/phone-list.m3u"
FILELIST=$(find "/music" -name "*.mp3")
TOTALNUMBER=$(echo "$FILELIST" | wc -l | tr -d ' ')
OLDIFS=$IFS
IFS=$'\n'
COUNT=0

INCLUDED_FILES[0]="/music/A Cappella"
INCLUDED_FILES[1]="/music/Acoustic"
INCLUDED_FILES[2]="/music/Alternative"
...
EXCLUDED_FILES[0]="/music/Progressive Rock/Spiral Architect"
...

for FILENAME in $FILELIST ; do
  for INCLUDE in "${INCLUDED_FILES[@]}" ; do
    if [[ "$FILENAME" == $INCLUDE* ]]; then
      for EXCLUDE in "${EXCLUDED_FILES[@]}" ; do
        if [[ "$FILENAME" == $EXCLUDE* ]]; then
          break 2
        fi
      done
      echo -n $FILENAME >> "/music/phone-list.m3u"
      echo -ne "\x00" >> "/music/phone-list.m3u"
      COUNT=$(( $COUNT + 1 ))
      continue 2
    fi
  done
  COUNT=$(( $COUNT + 1 ))
  echo -ne "    $COUNT/$TOTALNUMBER completed\r"
done
echo "    $COUNT/$TOTALNUMBER completed"
IFS=$OLDIFS
echo "Done. Playlist created at /music/phone-list.m3u"

Between the INCLUDED_FILES and EXCLUDED_FILES arrays, I was able to specify the prefix of any files/directories I wanted included/excluded.  Running the script produced the /music/phone-list.m3u file containing all of the paths to the files I wanted to transfer (null terminated).  Now, I could just feed this into rsync and kick off the file transfer:

rsync -avz --files-from="/music/phone-list.m3u" --delete -e "ssh -p 2222" -0 /. [email protected]:~/SDCard/Music/

And, of course, I can regenerate the file list and re-run the rsync command at any time to push any changes or new tracks over to my phone.

While playing back music on my phone, I've noticed a number of tracks missing cover art. Thought I'd share how I got that all fixed up.

First, I wrote a quick script to identify the list of files missing artwork:
#!/bin/bash
echo "Locating missing cover art..."
FILELIST=$(find "/music" -iname "*.mp3")
OLDIFS=$IFS
IFS=$'\n'
for FILENAME in $FILELIST ; do
    EYED3OUT=$(eyed3 "$FILENAME" | grep "FRONT_COVER Image")
    if [[ -n $EYED3OUT ]]; then
        continue
    fi
    EYED3OUT=$(eyed3 "$FILENAME" | grep "OTHER Image")
    if [[ -n $EYED3OUT ]]; then
        continue
    fi
    echo $FILENAME
done
IFS=$OLDIFS
Then, I would do a quick Beets ls query to make sure that the album search matched only what I wanted:
$ beet ls -a Circus Maximus Nine
When I was happy with the query string, I could feed it into the FetchArt/EmbedArt Beets plugins, as soon as I enabled them in the config.yaml for Beets:
...
plugins: fetchart embedart
...
$ ALBUM=Circus\ Maximus\ Nine && beet fetchart $ALBUM && beet embedart $ALBUM
If all goes well, the correct cover art should download and be embedded in the MP3 file. If for reason you need to update the cover art manually, just download it, and feed it to eyeD3:
$ eyeD3 --add-image ~/Desktop/cover.jpg:FRONT_COVER "/path/to/my/file.mp3"