rcFeedMe - A Fully Automated Podcatcher

rcFeedMe.zip (v1.41 - 11/5/09) (source is included)

- So I've been using Juice for a while as my podcatcher. It works ok most of the time but there were a few things that annoyed me: Sometimes downloads would just hang, no way to group feeds into different download directories, and sometimes something about a feed would change and all of the files would download. That last one might not be Juice's fault but still there are ways to solve this problem in the podcatcher.
- So I went looking to see if there was a better free alternative. Also, I'm one of those people who refuses to install itunes. I tried a few from this site:
http://www.podcatchermatrix.org/ but most of them had more issues than Juice with interfaces that seemed to make the whole process of configuring your feeds a chore.
- So I did what any programmer would do. I wrote my own. rcFeedMe is a podcatcher - basically it reads rss feeds and downloads them unattended all automatically on a schedule. It is NOT a rss reader. I use google reader for that. Once you set it up you shouldn't have to touch it anymore except when you want to add/remove feeds.
- Tip: If you find a show that only has an itunes feed and not a regular rss xml one read this article. I found it quite helpful.

Design goals/features:
- Quick and easy interface. It might not be that pretty but I wanted something that I could quickly enter settings for all aspects of the program.
- OPML import/export.
- Complete automation. Once setup, it just runs in the tray and checks the feeds for new downloads at an interval you set. No annoying popups or notifications. I just want it to silently do its thing. Downloads have to be as robust as possible and retry automatically if the download stalls.
- Maintain a history of every file downloaded so you will never get flooded with downloads when a feed changes or download something you've previously downloaded.
- Flexible system to determine where feed downloads are saved.
- Multi threaded with the ability to set the number of threads to use.
- Some basic abilities to sync to a mp3 player. I want to keep this program pretty focused on just downloading.
- Clean install. You just unzip the files and run so it's completely self contained and portable. All persisted data is stored in clear xml files.
- Source included

Developer Notes:
- Written in Visual Studio 2008 - c# targeting .net 3.5 framework. It took about a week to write the core functionality. It took another week to flesh out a few features and bug fixes.
- Utilizes Windows Communication Foundation (WCF) SyndicationFeed. I coded around their publish date parsing bug. Try it with ask a ninja and you'll see. SyndicationFeed isn't a very robust class and there were several feeds I had to massage the xml before it would load into a SyndicationFeed.
- I started with a pretty clean OO design but it broke down toward the end with me taking all sorts of shortcuts especially with presentation elements getting mixed in (mostly to help performance). This also means the memory footprint grew a bit too though cpu utilization is low.
- All persisted data is in clear xml files.
- Thread management is a bit of a mess. I'm sure there is a much better/cleaner way to do things.
- Overall life cycle of a file: First you subscribe to a feed (class Feed), I refresh that feed and get a list of all files (MediaFeed which is a keyed collection of Media; attribute on Feed). At this point in time I figure out what the filename will be and verify that a good link exists pointing to the file that will be downloaded. Next, the queue (class Queue which is a collection of Media) is built taking into account check keep counts and history. I then process the queue downloading all the files trying to spread download across different sites (this algorithm can be improved, it's very simplified right now). Once the entire queue is finished (or the user stops the downloads) I cleanup the queue. All successfully downloaded files are moved to History (collection of Media), all the ones stopped or errored are moved to errors (temporary collection of Media, never persisted to disk). Then I trim files based on keep count. It's a very safe and conservative trim. I walk the save paths resolved from Feeds (collection of Feed), I then search history by filename, if I find a match in history then it's considered for deletion based on save flag and keep count. Keyed collections were used instead of dictionaries because I needed serialization.
- I don't check free space anywhere so ummm yeah manage your disk space. The idea is keep count should keep space usage pretty constant.
- CleanupConfig.xml. This file is used to help cleanup the raw xml that causes issues with SyndicationFeed class. This has two sections in it. One is for search and replace, the other removes text between the start and end strings inclusive. In the future, as more problematic feeds are reported and fixed, hopefully I can just release an updated CleanupConfig.xml file instead of doing a new build. There is no gui for this file so if you want to manually edit it just use notepad.
- Other code used:
http://www.asp101.com/articles/christopher/filesystem/default.asp
http://www.brunofigueiredo.com/post/c-OPML-Parser.aspx (make sure you apply the fixes mentioned in the comments)
http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=176890&SiteID=1 - DataGridView progress bar
http://www.codeproject.com/KB/grid/DataGridViewColumnOrder.aspx - Save DataGridView column widths and position. Modified to persist settings in appSettings instead of user.config to keep rcFeedMe portable.
http://www.codeproject.com/KB/grid/CheckBoxHeaderCell.aspx?fid=452379 - Checkbox in DataGridView column header
http://www.veryicon.com/pack/aqua-smiles/ (icons)
- Thanks to coolmoedee345 for helping with the testing who listens and watches more podcasts than I think is humanly possible which of course made him the ideal tester.


Requirements:
- Windows XP, Vista, or Windows 7 (I've tested in both 32bit and 64bit)
- .net framework 3.5

Quick Start:
- Import an opml file or enter the url for the feeds you are interested in.
- Update the path on the settings tab to where you want to save your downloads.
- Click Start.

Usage:
- All the labels on the settings tab and many headers on the grids have tooltip text. That should help clarify some things.
- Command line parameters:
"1" or "auto" - (without quotes) to start in automatic mode. This means the app will minimize and the scheduler will start. This is useful as a shortcut in your startup group.
"2" or "lite" - The same as 1 except it puts it into "lite mode" which uses less memory but doesn't let you interact much at all with the program. Memory usage will still be pretty high when it's actually processing the download queue, but I try to free up as much resources as possible when rcFeedMe is idle and waiting for the next scheduled run.
"3" or "once" - Run Once: It's just like 1 except after finishing the run it will automatically exit. Useful if you would rather schedule rcFeedMe with windows scheduler and not have it running in the tray all the time.
- Right click on the grids for additional functionality. You can multi select rows in most grids by clicking the white space to the left of the row. Hold down ctrl to select multiple rows, shift for ranges of rows, and click the square in the top left corner to select all rows.
- Grids can be sorted by clicking on the header of the column.
- It should use proxy information that is setup for IE automatically though I haven't tested this.
- While a file is in the process of being downloaded it will have an extension of .rcFeedMeTemp. This extension is removed once the file has successfully completed downloading.

Main Menu:
File | Import opml: Import an opml file. Feeds with a url that already exists in the grid will be skipped.
File | Export opml: Export feeds to an opml file.
About: Shows the current version and a link to this post.
Debug | Save Settings: Not usually needed since Refresh All feeds saves all settings. Settings are also automatically saved when you close the application.
Debug | Load Settings: Not usually needed since Refresh All feeds saves all settings. Settings are automatically loaded on application start up.
Debug | Dump Raw XML Feeds: If you have a feed giving you problems (no files showing up/ERROR status when refreshing) you might want a raw dump of the rss feed. This will save the file in the same directory as the rcFeedMe.exe. It also dumps a fixed version if it's a problematic feed that I had to massage a bit to get working.
Debug | Sync Count: Count how many files that need to be synced and display it on the Sync button as Sync (n) where n = number of files that still need to be synced.

Toolbar:
Start: Start the scheduler.
Stop: Stop the scheduler.
Save then Reload Settings: Click this if you changed something and want to save it immediately but don't want to refresh all the feeds.
Refresh All Feeds: Saves all Settings, re-loads all settings, and refreshes all feeds. This is the button you want to click after you've added/removed feeds.
Process All Feeds: Does a Refresh All feeds, builds the queue, and processes the queue. This is the same thing that the scheduler runs.
Sync: Syncs files to a path of your choice (usually to a mp3 player or something). Paths are defined just like Save Paths but instead use the Sync Path fields (in feeds, groups, and settings). Sync is really a one way copy from where the file was downloaded to the Sync Path. Once a file has been copied, it is flagged as Synced in History so it won't sync again even if you delete the file on the mp3 player. You can always uncheck Synced in the history if you want to sync the file again. One small difference is sync paths are created if they do not exist (this includes even base directories). After the downloads are finished it will count how many files that need to be synced and display it on the Sync button as Sync (n) where n = number of files that still need to be synced. This lets me know in the morning if I have new podcasts waiting for me to be synced to my mp3 player. You can also get a count immediately with the Sync Count menu item under Debug.
Search History: Type in the textbox any substring you want to search in your history on Feed Name, Title, and File Name.

Status bar:
- First area is just general information on the last action.
- The next grey label area is the current overall status of rcFeedMe.
Idle - not much is happening.
Feed - Feeds are in the process of being refreshed.
Queue - Queue is being processed. Files are being downloaded.
Sync - Files are being copied to your mp3 player.
You will see the word [Lite] next to the status if you are running in low memory usage Lite mode.
Depending on the status, certain functions might not be available.
- Progress Bar shows status of either the feeds refreshing or the download queue. Download progress is only updated every percentage change.
- The tray icon will show teeth while rcFeedMe is being fed (downloading). Actually, it's anytime it is not idle.

Feeds Tab:

At the top is a grid displaying all your rss feeds and a variety of parameters to customize how files are downloaded. At the bottom displays all the files associated with the feed you clicked on in the top grid. Note: You must click "Refresh All Feeds" before you will see anything in the lower grid.
- How the save path of a file is determined: If a path is specified at the feed level, then that exact path will be used. Next, if a group is specified, then the group path will be used as the base path, a sub directory with the feed name will be created in this base path, and then the file will be saved in that sub directory. Finally, if a feed has no save path defined, group is set to (None), then the Save Path on the Settings Tab will be used as the base path. The same sub directory logic is used as in the group scenario. Note: Sync Path follows the same logic.
- Please make sure all base directories are created first or the program will throw an error.

Feeds Grid (top):
- Name: The name of the feed. Please be careful when editing this field. It's best not to touch it once you've begun downloading files for the feed since I use feed name + file name as the key to determine if something has already been downloaded (history).
- Url: The address that points to the feed. If you enter this first and leave Name blank then it will try and fetch the name of the feed for you. It's not quite as robust as refreshing the feed so if this fails, try typing the feed name manually and then refreshing the feed. There is a chance it still might work.
- Group: Any groups defined on the group tab will be in this drop down. Assign this feed to a group or leave it set to (None).
- Keep Count: This number determines two things. 1) It determines the maximum number of possible files that can be downloaded for the feed. So if it is set to 5 then at most 5 of the most recent files will be downloaded for this feed. 2) It also determines the maximum number of shows to keep on your hard drive. So if you download the latest 5 files, then a new one shows up in the rss feed, then the new show will be downloaded and the oldest existing file will be deleted. This is the primary method of managing disk space. By specifying keep counts your disk space used shouldn't change much. Set it to 0 to disable the feed. Enter -1 if you want to download and keep everything. I've seen other podcatchers use number of days or something like that which I find inconvenient since different feeds publish new shows at different or irregular intervals.
- Include Extension: Pipe(|) delimited list of file extensions(include period(.)) to match against for downloads. Sometimes undesirable file types get included into the feed (say a .aspx or something) or the feed is a consolidated feed with the same show in different file formats (say wmv, mov, avi, etc). This lets you filter down the files to download by a list of extensions. Example: if you only want wmv files to download from the feed then enter ".wmv" (with the period but no quotes, search is case insensitive). If you wanted wmv and mov files then enter ".wmv|.mov".
- Include Text: pipe(|) delimited list of substrings that are compared to the filenames for download. This is similar to extension but is a substring search of the entire filename. If it finds a matches then the file will be downloaded. This is useful if a feed contains many different versions of the same show in the same file format but at different quality levels/bit rates. Example: a feed has Show01High.wmv,Show01Med.wmv,Show01Low.wmv. If you wanted just the high quality ones you would enter "High" (without the quotes, search is case insensitive).
- Save Path: Enter the complete path where the podcasts for this feed will be saved. This path overrides all other paths (groups and global settings).
- Prepend Date? Prepend the Publish Date to filename. Useful to make the filenames unique or help with sorting. Date format used is YYYYMMDD so it sorts propery. If both Prepend Date and Prepend Title are checked, then Date comes first.
- Prepend Title?: Prepend title to filename. Useful to make the filenames unique if only a generic filename is used in the feed. Take a look at the GameTrailers feeds. All the filenames are just download.mp4. Checking this will take the title of the show and prepend it to the filename. So download.mp4 will become "show title - download.mp4".
- Sync Path: Just like save Path but defines where files are copied to when you click the Sync button.
- Sync?: Check if you want files from this feed to sync to your mp3 player.
- Apply Cache Fix?: Appends a fake query string to the end of the url when making the request for the feed. This guarantees a cached copy will not be used but also causes some sites to throw an error. Most sites don't need this. If you do check this, then do a refresh and make sure the feed still works. To be honest I'm not sure if this "fix" really helps anything. I notice on a few feeds (like hdtv podcast) where there is a bit of a lag before the latest show appears in the feed. What is odd is if you view the feed in a browser (and therefore the xslt is applied) you see the latest version of the rss feed. If you instead grab the raw xml (I use webClient but trying httpwebrequest doesn't make a difference) it's the previous version and takes a while before it's updated. If I use WebBrowser class then yeah I get the latest data but it's after the xslt is applied so it's html and no longer rss xml. It's pretty odd. It only happens on a few feeds, and it's not a big deal since the raw rss xml eventually does get updated. If any programmers out there can explain to me what is happening, why, and maybe how to fix it that would be great.
- Right Click menu lets you abort refreshing feeds.

Files Grid (bottom)
- Add to Queue: Manually add the file to the download queue. The file will be automatically flagged as "Keep". See History Tab for more information.
- In Queue?: Read only - checked if the file is already in the queue.
- Title: Show title. This is the text used in the filename if Prepend Title? was checked above.
- Summary: Show summary
- Publish Date: Date the show was published. This is the date used in determining the latest shows to download. By default the grid is sorted descending on this field.
- File Name: Filename that will be used when downloading and saving the file.
- Link: The url pointing to the actual file that will be downloaded.

Download Queue Tab:

- I attempt to spread the downloads across different feeds to help maximize speed and thread efficiency.
- Feed Name: Name of the feed.
- Title: Show Title
- Link: The url pointing to the actual file that will be downloaded.
- File Name: Complete path and filename of what is being downloaded and saved.
- Publish Date: Date the show was published.
- Status:
Queued - In the download queue but has not started downloading yet.
Downloading - Download is in progress.
Downloaded - File was downloaded successfully.
Error - File was not downloaded properly. Some sort of error occurred. Partially downloaded files in error status are automatically deleted.
Stopped - User right clicked on the Queue grid and selected "Stop All Downloads". Any downloads in progress will be stopped immediately and have this status. The partially downloaded file will be deleted.
Bytes - Number of bytes currently downloaded/Total Bytes.
Progress - % downloaded
Attempt - How many times rcFeedMe tried to download this file. Once attempts hits the download retry maximum (set in the Settings tab) then the file will be set to Error status.

Download Queue Context Sensitive Menu (right click):
- Build Queue: Not used often unless you want manual control over some of the steps that "Process All Feeds" does. Click this after doing a Refresh All Feeds. It will build the queue but not actually start downloading yet.
- Process Queue: Begins downloading the files in the queue.
- Stop all Downloads: Immediately stop all downloads and halts the processing of the queue.
- Remove Selected Rows: Remove the files from the download queue.
- Move Selected Rows to History. This is more for debug but it might come in handy. After build queue you can move rows from the queue to History. This will trick the program into thinking these files were already downloaded. Useful in some situations where you want to pre-populate the history. Maybe there is a specific show you just don't want to download.

Download History Tab:

- After the queue is completely finished processing, all successfully downloaded files are moved here. The history will continue to grow over time. It might be a good idea to trim it down once in a while by sorting by date and then selecting multiple rows of the oldest data and deleting it.
- I won't go through all the columns since I've already describe most of them by now. Three columns are editable on this grid: Title, File Name, and Keep?.
- I allow editing of Title and File Name since these combined is the primary key I use to determine if a show has already been downloaded. This is just in case somebody wants to tweak something in the history.
- Keep?: Checking this prevents rcFeedMe from deleting the file. This goes along with Keep Count back on the feed tab. Any files you added manually from the files grid on the feeds tab will have this already checked. It makes sense because you added the file manually (and not by the scheduling engine) so the scheduler shouldn't delete it. Kept shows are excluded from the keep count calculation.
- Right click | purge history: This will clear the entire history file (a backup is created though just in case). Useful if you want to reset and re-download
everything.
- Synced?: If checked it means the file has already been synced to your mp3 player so it won't get synced again in the future. I you wish to sync the file again for some reason just uncheck this box.
- Make sure you save any changes made before leaving this tab since this tab gets refreshed whenever you click on it.

Recent Errors Tab:

- After the queue is completely finished processing, all the files that failed to download are moved here. This only shows files that error out on the most recent run. The data in this tab is not persisted anywhere so if you close rcFeedMe and then restart it, it will be blank. If you want to see a running history of all the errors from this tab, please see Errors Log Tab.

Group Defaults Tab:

- Specify the download path for the group. The groups you define here appear in the drop down on the Feeds tab.

Settings Tab:

These are your global preferences.
- Frequency: This is how often the scheduler should refresh all feeds and download any new files. Default is 360 minutes (every 6 hrs).
- Default Save Path: If a feed doesn't have a path specified and group is set to none, then this path will be used as the base path. See Feeds Tab for a more detailed explanation of how the download path is determined for a file.
- Default Sync Path: If a feed doesn't have a path specified and group is set to none, then this path will be used as the base Sync path.
- Keep Count: Default value used to populate the Keep Count field on the feeds tab when importing an opml file. Default is 5. See Feeds Tab for more information.
- Threads used to Scan: Number of threads to use when refreshing the rss feeds. Default is 6. Maximum allowed is 16.
- Threads used to download: Number of threads to use when downloading files. I would recommend 2. Any higher and some sites limit the number of simultaneous downloads and could produce false failed downloads. I try to minimize this by spreading out simultaneous downloads across different feeds. Maximum allowed is 6.
- Scan Stall: Number of seconds that elapse of inactivity while a feed is being refreshed. If the feed refresh stalls for this many seconds, it will be considered an error.
- Download Stall: Number of seconds that elapse of inactivity while a file is being downloaded. If the download stalls for this many seconds, it will be considered an error.
- Scan Timeout: If the total time for a feed refresh takes longer than this many seconds, it is considered an error.
- Download Timeout: If the total time for a file download takes longer than this many seconds, it is considered an error.
- Scan Retry: When an error occurs during a feed refresh, retry this many times before giving up.
- Download Retry: When an error occurs during a file download, retry this many times before giving up.
- Time Zones (Advanced): Used to help parse Publish Date. Sometimes it helps to strip out the time zone.
- Date Formats (Advanced): Used to help parse Publish Date. Specify the .net Custom Date and Time Format string. see: http://msdn.microsoft.com/en-us/library/8kb3ddd4.aspx

Log Tab:

- A very detailed log of everything that happens in the program. The log self trims so it won't grow in size forever. The log is refreshed every time you go to this tab or you can right click and refresh immediately. The log will grow until it hits 1.5MB. After that the oldest entries will be deleted and the log will be automatically trimmed down to 1MB. This cycle just keeps repeating. Log size check and trimming occurs when you process your queue.

Error Log Tab:

- Since the Recent Errors Tab isn't saved anywhere I decided to create a separate error log. The information is also buried in the complete log but this log only contains information that was in the recent errors tab. This log also self trims.


Hopefully you find this program useful. Leave any questions or suggestions in the comments, and I'll try to respond.

UPDATE: If you have issues with a specific feed give http://feedvalidator.org/ a try. It's quite helpful in pointing out where the problem might be.

Version History:
v1.41 (11/5/09)
- This is mainly user interface tweaks. All checkbox columns now have a checkbox in the column header so you can easily toggle the checkboxes for all rows.
- All grid column positions and widths are now saved. Even the split location on the first tab is saved. All these settings are saved in rcFeedMe.exe.Config. So if you want to reset just delete the appSettings section.

v1.40 (11/2/09)
- added httpWebRequest useUnsafeHeaderParsing="true" to rcFeedMe.exe.config. This fixed an issue with a specific feed. You can read more about the issue here.
- cleanupconfig.xml is now a lot more flexible. You can now add fixes that only apply to a particular feed url instead of globally to all feeds. This allows me to add specific feed fixes without the risk of breaking some other feed. Just add the optional attribute feedUrl to either ReplaceStruct or RemoveStruct. If you leave feedUrl attribute off then it still applies globally.
- Added Restore to the right click menu on the tray icon. It does the same thing as double clicking the tray icon.

v1.39 (4/21/09)
- updated CleanupConfig.xml to handle another problem feed. Fixed a rare issue where if a filename differed only in case it would hang the feed refresh process.

v1.38 (2/22/09)
- Fixed a bug where I was never using DL stall time from the config (I was accidentally using scan stall). Also, rcFeedMe no longer shows up in the alt-tab list when minimized to the tray.

v1.37 (2/16/09)
- Fixed a problem with idle thumbs feed. Added better error logging when there is an xml parsing error. I now try and extract the xml fragment that was causing the problems and save that into the log.

v1.36 (1/27/09)
- Some feeds are missing the type attribute on the enclosure tag. I've added some code that should work around this issue.

v1.35 (1/11/09)
- added new Error Log tab. Anything that ever shows up in the Recent Errors tab is archived in this log. Sure the same information is also captured in the compete Log but this is easier to read if you just want to look at your failed downloads. This log also self trims.

v1.34 (1/7/09)
- added a size check. When media is finished downloading, I check received bytes against total bytes expected. If it received less then it is considered an error.
- latest CleanupConfig.xml is rolled into this release

v1.33 (12/12/08)
- fixed a tiny bug where I forgot to reset the tray icon tool tip text and form title after doing a sync.

v1.32 (12/11/08)
- Files that are too long are now trimmed down. This can happen if you prepend the Title and it happens to be a very long title. I've set maximum file name length to be 240. It will be less than this since I subtract out the full path. Any filenames that needed trimming will show up in the log.
- Fixed a bug when deleting rows out of the Groups tab.

v1.31 (12/10/08)
- Improved error messages in the log if a download fails.
- Sync Count now also displays in the window title and the tooltip text for the tray icon.

v1.30 (12/03/08)
- added a new feature: Sync Count. After the downloads are finished it will count how many files that need to be synced and display it on the Sync button as Sync (n) where n = number of files that still need to be synced. This lets me know in the morning if I have new podcasts waiting for me to be synced to my mp3 player. You can also get a count immediately with the Sync Count menu item under Debug.

v1.29 (12/03/08)
- fixed a bug introduced in the new feature in v1.27 and made worse in v1.28: Crash bug when editing existing feeds.

v1.28 (12/02/08)
- fixed a bug where if you had no group assigned to the feed, clicking add to queue button would cause the app to blow up.
- fixed a crash bug where feed name was null when it is trying to fetch the name.
- added a new xml file: CleanupConfig.xml. This file is used to help cleanup the raw xml that causes issues with SyndicationFeed class. This has two sections in it. One is for search and replace, the other removes text between the start and end strings inclusive. In the future, as more problematic feeds are reported and fixed, hopefully I can just release an updated CleanupConfig.xml file instead of doing a new build. There is no gui for this file so if you want to manually edit it just use notepad.

v1.27 (12/01/08)
- When subscribing to new feeds, if you enter the url first and leave the Name blank, then it will try and fetch the name of the feed for you.

v1.26 (11/24/08)
- fixed some problem feeds that use dc:date instead of pubDate tag in the rss xml.

v1.25 (11/21/2008)
- Initial public release