Throughout this document, the series of scripts that is HDHR DVR Video Process will be referred to as VP. At the moment, the scripts work for Linux only and has only been tested on Ubuntu 14.04.
VP requires python 3 and is written entirely in python 3.4. Throughout the scripts, several Linux programs are called using bash using pythons subprocess. This requires them to be installed before use of VP. These programs are as follows:
- CCextractor - program to extract the closed captioning and turn it into usable subtitles
- Mediainfo - program to quickly get information about a program
- FFMPEG - video transcoder for many codecs. Using version 2.2 because it is known to work with comskip
- ProjectX - Java based application to demux the video and audio into elementary streams while also doing some error checking
- MKVtoolkit - Mkv suite used to mux all the files into one before transcode
- Comskip - checks entire video files for indications of a commercial and notes it into an edl file. Making use of the linux port
- HandBrake - fast modern video transcoder used for the x.264 codec
- run-one - makes sure only one instance of a cronjob is run at each cronjob call
The zip file containing VP has a recommended folder structure for your HDHR DVR. There is a text file in /convert/tools/config/ called “config.txt” This document needs to be updated to suit the location of your VP folder. Where it says /yourdrive/ in each line you should replace yourdrive with the full path to the VP folder. I chose to call the containing folder “DVR” and placed it in my Video Library on the Same level as “TV Shows” and “Movies” you can place it where it makes the most sense for you.
This document can be updated to reflect where you want things to go and end up as well. Some are named properly but here is a thorough description of each location.
recordings - the location of where you told HDHR to record your programs
work - a location where files are held while VP is processing a program
tools - location of all script files and some config files
grave - location of original recording files held after processing
subname - text file used for cases where HDHR DVR program names do not match thetvdb
Some programs do not match thetvdb with whatever tv scheduling information is used by HDHR DVR. To bypass this you can use the helper file specified by subname(/convert/tools/configmanualmatch.txt). You simply write the name used in the recordings folder followed by an equal sign and then the exact name intended that is found in thetvdb.com. Otherwise, VP will try to find an exact match and if not found use the name and season info given by HDHR DVR. Do note that VP checks the subname file before the blind exact match.
tvout - location of processed tv shows that have successfully been matched with thetvdb.com
movieout - location of processed movies that have successfully been matched with imdb.com
manualout - location of processed tv shows and movies that have not been matched with a database
tempout - location used when HandBrake is currently transcoding a file
logarchive - where old hdhr log files are kept
cleanlib - a .txt file that specifies the shows wanted to be limited by most the 6 most recent
library - specifies where the TV show library is
A compliment script included with VP can be used called simply clean.py. It is meant to be used daily and has three functions:
- Archiving the log file the HDHR DVR creates daily
- Delete original files and conversion files
- Delete converted files in your library
The clean script is used archive/delete unused programs and files. It first puts the old log files into a specified folder. It then deletes all original recordings that are older than 2 days. It then looks at the cleanlib file (/convert/tools/config/dailyshows.txt)for a list of series in your library keeps the 6 most recent programs, and deletes any others. This is only intended for libraries that have a strict “/Show Name/Season ##/Show Name.S##E##.Episode Title” unintended things may happen if used on a differently structured library. Simply add the show you want to limit to 6 most recent kept episodes to this file.
The hdhomerun.conf file should route the recordings to the VP “recordings” folder. Make sure you change it to your particular recordings folder. This means when you start up hdhomerun_record you must specify the conf file like so “/DVR/app/hdhomerun_record start --conf /yourdrive/DVR/convert/tools/hdhomerun.conf”
To install VP simply extract the contents of the DVR folder into a new folder on the system you have HDHR DVR running on. This folder will now contain the dvr app. I found DVR to be an appropriate folder name but you can name it whatever you'd like. Next, change the config.txt file described above to have full pathnames to where you put the folder. Now you must also change the homrundvr.conf file to have a fullpath to the recordings folder also described above. Now you can startup the hdhomerun_record app as described above.
VP program sequence
To give you a better idea of exactly what is going on when VP is run here is a list of what is happening when processlist.py is called.
- a list of all recordings is created with the .mpg file ending, and all current recordings are removed from that list and each recording is cycled through and startprocess.py is called on the first recording.
- based on the show name, season number, episode number, or movie name the recording is given a new name checked with thetvdb and a number of "-" separated parameters. 1 for movie, 2 for a show not checked with thetvdb and 3 for a show checked with thetvdb. with this new name it is passed to the notranscom.py.
- it is then given a folder name and a log file located in the grave where the original file is copied to.
- notranscom.py calls on ccextrator to extract the closed captioning and translate it into a usable srt file
- next projectx is run to separate the main video and audio streams (demux) from the original recordings. ProjectX does some error checking for damaged recording streams
- now mkvmerge remuxs the video and audio streams so Comskip can run on the new file. Comskip outputs an "edl" file containing the parts of the show that have commercials.
- using this same edl file a chapter file is created at the end of each commercial break.
- finally using the video, audio, subtitle, chapter, and edl files a new file is created using mkvmerge with the commercials comskip found removed.
- this file is then passed to trans.py where it is given a new name with period separators and using handbrake it is transcoded to the h.264 codec, detelcined, and reduces to or maintains 720p. the finished file is then moved to a new folder for further sorting based on if it is a movie (movie folder), confirmed thetvdb show (tv folder), or unconfirmed thetvdb show (manual folder)
Running VP manually
To run clean call “python3 /DVR/convert/tools/clean.py /DVR/convert/tools/config/config.txt”
The app folder should contain the hdhomerun_record file obtained directly from the silicondust forums. It will also contain a “log-archived” folder for older log files.
You can run VP a couple of different ways currently. First, to fully process a single video call “python3 /yourdrive/DVR/convert/tools/startprocess.py /DVR/convert/tools/config/config.txt”. Second, to process all programs in the recordings folder call “python3 /yourdrive/DVR/convert/tools/processlist.py /DVR/convert/tools/config/config.txt”.
Running VP using cron (recommended)
Make sure you have installed run-one so that you don't end up trying to transcode multiple recordings at once
To run VP every half hour add this to your crontab "10,40 * * * * run-one /usr/bin/python3 /yourdrive/DVR/convert/tools/processlist.py /yourdrive/DVR/convert/tools/config/config.txt" with the highest privileges. This will run every half hour at minutes 10 and 40. 10 minutes after each show starts because between real time and the HDHR DVR log file (which updates every 2 minutes) there is a 2-minute delay. The remaining 8 minutes allows for approximately 3-4 log file updates, enough to tell which programs are currently recording.
To run the clean script every day add this to your crontab "3 20 * * * /usr/bin/python3 /yourdrive/DVR/convert/tools/clean.py /yourdrive/DVR/convert/tools/config/config.txt"with the highest privileges. This will run clean at 8:03pm EST. For eastern time, this is great because it is at UTC time 0:03 just after the newest log file is created so any older log files are sent to the archive folder. This ensures that only one log file is kept in the recordings folder at once. I recommend finding out what the UTC time 0:00 is for your time zone and updating the cronjob. Be sure to update your library in the config file and also clear out the cleanlib.txt file and add shows you want to limit the accumulation of.