Skip to content

Instantly share code, notes, and snippets.

@asifbacchus
Last active December 17, 2018 05:56
Show Gist options
  • Save asifbacchus/39caf6b2ce88bf208b346a9cb9c2abdd to your computer and use it in GitHub Desktop.
Save asifbacchus/39caf6b2ce88bf208b346a9cb9c2abdd to your computer and use it in GitHub Desktop.
Starting around December 2018, Vader Streams' EPG cannot be read by TVHeadend. This is due to various <!ELEMENT xx> tags at the beginning of the file. Attached is a script to resolve this problem and an explanation of how it works.
#!/bin/sh
#######
### Download Vader EPG XML file and remove the <!ELEMENT> tags that TVHeadend
### does not understand.
#######
### This script was written by Asif Bacchus ([email protected]) and comes with
### no warranty, guarantee or promise of suitability for any purpose whatsoever.
### Use this at your own risk and understand that doing so indemnifies the afore
### mentioned author from any and all liability.
# Last modified: December 12, 2018.
### Define output formatting ascii codes
err="\033[1;31m"
note="\033[0;96m"
bold="\033[1m"
normal="\033[0m"
### Get name of THIS script
scriptName="$( basename ${0} )"
### Check that command-line parameters have been provided, else exit
# Check if a download URL has been provided to this script
if [ -z "$1" ]; then
printf "%b\n" "${err}Please provide the download URL for the EPG XML file.${normal}"
printf "%b\n" "${note}Usage: ${scriptName} http://download.url/file.xml outputFile.xml${normal}"
printf "%b\n" "${bold}Exiting now.${normal}"
exit 1
else
printf "%b\n" "${note}Downloading from: ${1}${normal}"
fi
# Check if a filename was provided for the edited EPG XML list. If not, exit.
if [ -z "$2" ]; then
printf "%b\n" "${err}Please provide a filename for the edited EPG XML file.${normal}"
printf "%b\n" "${note}Usage: ${scriptName} http://download.url/file.xml outputFile.xml${normal}"
printf "%b\n" "${bold}Exiting now.${normal}"
exit 2
else
printf "%b\n" "${note}Saving output to: ${2}${normal}"
fi
### Download EPG from Vader Streams and save as desired filename
# download file
printf "%b\n" "${note}Downloading...${normal}"
wget --quiet "$1" --output-document "./$2"
# check if download was successful, exit if not
checkResult="$?"
if [ "$checkResult" -gt 0 ]; then
printf "%b\n" "${err}There was a problem downloading the EPG file.${normal}"
printf "%b\n" "Please verify you have the correct URL and try again."
printf "%b\n" "${bold}Exiting now.${normal}"
# delete the generated output file
rm -f "./$2"
# exit with error code 100
exit 100
fi
### Edit the file in-place to conserve space on embedded devices
# remove problematic tags in the 'tv' section at the beginning of the file
printf "%b\n" "${note}Removing problematic <!ELEMENT xx> tags${normal}"
sed -i '/^<!-- DTD/,/^]>/{/^<!-- DTD/!{/^]>/!d}}' "$2"
# remove any extraneous tags on the 'DTD' line (formatting mistakes)
printf "%b\n" "${note}Cleaning up DTD line${normal}"
sed -i 's/xmltv.dtd -->.*/xmltv.dtd -->/' "$2"
### Exit cleanly
printf "%b\n" "${bold}Success! Exiting now.${normal}"
exit 0

Fixing Vader Streams EPG for use with TVHeadend (TVH)

The problem

Since early December 2018, Vader Streams has changed their EPG xml file. This new file includes several tags which are apparently unsupported by the TVH EPG-grabber/parser. Trying to use this downloaded file directly will result in an error that reads htsmsg_xml_deserialize error Unknown syntatic element: <!ELEMENT tv where 'tv' can be any of several tags.

The workaround

The offending tags must be removed so that the XML file can be parsed by TVH. These tags are ALL at the very beginning of the file. They immediate follow the <!-- DTD for TV... and stop at the ]> line. Here's a sample:

<?xml version="1.0"?>
<!DOCTYPE tv [
<!-- DTD for TV listings http://xmltv.cvs.sourceforge.net/viewvc/xmltv/xmltv/xmltv.dtd -->
<!ELEMENT tv (channel* , programme*)>
<!ATTLIST tv date CDATA #IMPLIED>
<!ATTLIST tv source-info-url CDATA #IMPLIED>
<!ATTLIST tv source-info-name CDATA #IMPLIED>
...
<!ATTLIST review reviewer CDATA #IMPLIED>
<!ATTLIST review lang CDATA #IMPLIED>
]>
<tv source-info-name="" source-info-url="" source-data-url="" generator-info-name="jaylinski/xmltv" generator-info-url="https://github.com/jaylinski/xmltv">

After removal, the beginning of the file should look like this:

<?xml version="1.0"?>
<!DOCTYPE tv [
<!-- DTD for TV listings http://xmltv.cvs.sourceforge.net/viewvc/xmltv/xmltv/xmltv.dtd -->
]>
<tv source-info-name="" source-info-url="" source-data-url="" generator-info-name="jaylinski/xmltv" generator-info-url="https://github.com/jaylinski/xmltv">

A file formatted like this will work in TVH as normal and will populate your EPG as before.

Automating it -- using a shell script

Manually doing this to your downloaded EPG is an option, but not a very good one. We can use a simple script instead to download the file, clean it up and put it somewhere TVH can find it. I've tried to make the script POSIX compliant so it can be run on most any flavour of LINUX.

Using the script

  1. Log in to your system via SSH or at the console.
  2. Download/copy the script to your machine. You can rename the script to anything you like, it doesn't matter.
  3. Make the script executable
chmod +x scriptname.sh
  1. You should run the script once manually to make sure everything works properly.
  2. The script takes two arguments in order to run. If you forget an argument, the script will exit and display a brief usage reminder so you know what you did wrong. The syntax is:
scriptname.sh http://server.tld/filename.ext outputFile.xml
  • scriptname.sh -- Whatever you named the script (default is downloadEPG.sh)
  • http://server.tld/filename.ext -- This is the URL where the EPG file is located. Currently, for Vader Streams, this is http://vaders.tv/p2.xml
  • outputFile.xml -- This is whatever you want the edited XML file containing the EPG to be called. I usually call mine epg.xml. You can supply a full path or just a filename if you want the file saved in the same directory as the script.
  1. Assuming everything runs correctly with no errors (the script will tell you), you should have an edited file you can use with TVH.
  2. If you're using KODI, update your TVH service to point to the file you just created. If you're using something else or using TVH to directly access the file, update as necessary to reflect the new filename.
  3. That's it! It should work for you just as it did before :-)

Setting up a cronjob to run the script automatically

On pretty much every Linux system, you can set a cron job to run this script and keep your EPG data up-to-date. Here are the steps:

  1. Make sure you're logged in as root or some other user that is ALWAYS logged in since you need this to run unattended.
  2. Open your crontab to create/add a new job
crontab -e
  1. Go to the bottom on the file, add an empty line or two and then add the following: (If your file is empty, that's cool, just add the following)
# Download EPG from Vader Streams at 2:22am every day
22 2 * * * /storage/downloadEPG.sh http://vaders.tv/p2.xml epg.xml >/dev/null 2>&1

Of course, you'll have to customize this to fit your environment. Here's a brief explanation to help you out:

  • Edit the comment to say whatever you want. That's the line that starts with #
  • "22 2 * * *" means "run every day at 2:22am". If you wanted to run the script at 3:45am, you'd use "45 3 * * *". If you wanted 6:13pm, then you'd go with "13 18 * * *". In other words, it's minute <space> hour <space> then three asterisks. Those three asterisks mean every day of every month each day of the week. Check out https://crontab.guru if you need help making an expression for the time/days you want to run the script.
  • My script is located at /storage/downloadEPG.sh since I'm running on LibreElec. Again, you can store the script anywhere that works for your environment.
  • Pass the appropriate parameters just like when you ran it manually before.
  • >/dev/null 2>&1 means that you don't want any output from the script and no notifications from it. Since it's running unattended, this makes sense. You can copy this part verbatim.
  1. Save and exit. Accept whatever filename is automatically generated, do NOT change it.
  2. Verify your crontab has been updated:
crontab -l

You should see your changes appear.

  1. That's it, your script is set to run at the time you specified.

Make sure you log back into your system via SSH/console the next day or two and check your downloaded EPG file. You should see that the timestamp of the file has been updated, indicating that a new file was downloaded and everything is working.

Final notes

I think that's it. Hopefully, this was pretty clear. If not, leave a note below and I'll get back to you.

@asifbacchus
Copy link
Author

This problem seems resolved on Vaders' end after EPG maintenance. I'll keep this available for a bit longer in case things revert.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment