I can see the lineup created by BigScreenEPG in my guide editor using the SchedulesDirect data, which even includes appropriate cablecard channel numbers if the "Include Channel Numbers in Output" checkbox is checked, but it isn't replacing the existing primary lineup with the BSEPG lineup, so the actual MCE guide continues to use Rovi (or empty channels numbered 1-9999 if I do TV setup without allowing it to download guide data first). This seems a lot closer than MCImportXML which is missing both channel numbers (so everything would need to be mapped by hand) and HD flags by default.
I tried to update the primary lineups list programmatically, and it seems I can add to it, but attempts to remove the Rovi lineup fail with an error saying RemoveAllMatching only works with lineups that are members of a Lineup Collection in an object store or something along those lines.
The call to MergedLineup.FullMerge to presumably update the actual channel list in the guide fails with access denied.
I suppose I could write something to loop through all the channel numbers in the BSEPG lineup and find the corresponding guide channels and update the listings for each of the 900 channels individually, but
a) I'd rather use something already proven, and it is really odd to me that BSEPG is not doing it by default.
b) if it works, this would need to be rerun periodically whenever Verizon rearranges their channel list, or at least continue manually updating whatever channels they change.
Got Zap2IT working on WMC! with a LOT of work
-
glugglug
- Posts: 391
- Joined: Thu Jun 09, 2011 1:34 am
- Location:
-
HTPC Specs:
-
glugglug
- Posts: 391
- Joined: Thu Jun 09, 2011 1:34 am
- Location:
-
HTPC Specs:
I went and wrote an app to replace all the channels in the guide with the ones from the BSEPG listing in bulk, using all available CableCARD tuners. If you aren't using cablecard, the results won't be desirable -- the channels are created using tuners associated with the "Scanned (Digital Cable (CableCARD™))" lineup, so if this doesn't exist you'll be populating your guide with channels that can't be tuned, if it doesn't just crash.
Seems to have worked! Going to leave it running for the rest of my 1 week SchedulesDirect trail on my spare machine and see if it continues to work with the 2 scheduled tasks (1 for mc2xml, one for BSEPG). Will probably upload it this weekend.
Seems to have worked! Going to leave it running for the rest of my 1 week SchedulesDirect trail on my spare machine and see if it continues to work with the 2 scheduled tasks (1 for mc2xml, one for BSEPG). Will probably upload it this weekend.
-
glugglug
- Posts: 391
- Joined: Thu Jun 09, 2011 1:34 am
- Location:
-
HTPC Specs:
Created an app to clone the lineup created by BSEPG or other tools into your guide: http://www.thegreenbutton.tv/forums/viewforum.php?f=9
Requires channel numbers, so make sure it is set to include them.
I suspect you also probably want to NOT use channel matchnames. Unlike the matchnames, the matching by number won't get any wrong that you need to manually fix.
Requires channel numbers, so make sure it is set to include them.
I suspect you also probably want to NOT use channel matchnames. Unlike the matchnames, the matching by number won't get any wrong that you need to manually fix.
-
audinutt
- Posts: 114
- Joined: Thu May 14, 2015 2:43 am
- Location:
-
HTPC Specs:
glugglug wrote:Created an app to clone the lineup created by BSEPG or other tools into your guide: http://www.thegreenbutton.tv/forums/viewforum.php?f=9
Requires channel numbers, so make sure it is set to include them.
I suspect you also probably want to NOT use channel matchnames. Unlike the matchnames, the matching by number won't get any wrong that you need to manually fix.
http://www.thegreenbutton.tv/forums/vie ... f=9&t=9324
Link to tool page.
-
audinutt
- Posts: 114
- Joined: Thu May 14, 2015 2:43 am
- Location:
-
HTPC Specs:
Update.. I have been using a script I wrote to kill all wmc processes and restore an old epg database once a week... this allows two weeks of guide data and purges the generic descriptions that otherwise get stuck in the epg. I will post the script if anyone is interested. I am also using syncback free to handle the backup and restore of the epg database.
-
audinutt
- Posts: 114
- Joined: Thu May 14, 2015 2:43 am
- Location:
-
HTPC Specs:
The zap 2 it epg has been working really well with this final tweak.
-
dublus
- Posts: 29
- Joined: Fri Apr 05, 2013 5:47 pm
- Location:
-
HTPC Specs:
Has this thread died out?
I was hoping for a progress update and instructions on how to implement the script(s) mentioned. The lack of complete guide info is really starting to get to me.
Is it possible audinutt and glugglug could put their collective methods, apps, scripts together to make something that works "as well as possible" with the least room for user error?
I was hoping for a progress update and instructions on how to implement the script(s) mentioned. The lack of complete guide info is really starting to get to me.
Is it possible audinutt and glugglug could put their collective methods, apps, scripts together to make something that works "as well as possible" with the least room for user error?
-
glugglug
- Posts: 391
- Joined: Thu Jun 09, 2011 1:34 am
- Location:
-
HTPC Specs:
Working on an app, but it is pretty complicated...
One thing I figured out is that XMLTV is far from being a complete specification. As such, things are kludged in to try to work around it, and none of the XMLTV producers includes even close to the complete data set that you can download through the SchedulesDirect JSON API. The specific properties/enumerations that are in the API and the MXF format are obvious parallels BTW -- the Media Center format was **made for** Zap2It data. (Which is still a subset of what SchedulesDirect JSON API has, but XMLTV only supports a very small subset of that, and which subset is inconsistent from one grabber to another since everything about it is kludgy).
Other things:
The MXFImport/MXFExport APIs to load data into WMC are slow and memory intensive even on a fast machine. Just calling the function to make Media Center create a complete MXF of what is currently in your WMC config uses about 6GB of RAM and takes about 15 minutes (or maybe that is because I have both the Rovi lineups the BSEPG lineup in it). The MXF I am creating so far with SD data is more than double the size of the BSEPG one since it has more detail by not going through XMLTV and I haven't even added the schedules or Channels to it yet. Currently has program info (including cast/crew, both long & short descriptions, image URLs), series info, keywords, and keyword groups.
Never found a way to get WMC to create the channels itself from the MXF import, as it does with its own lineups. I am downloading the list of stations/channels for each lineup, and figuring out based on your configured tuners a set of default channel tuning infos for each station. Haven't done the UI for overriding these defaults yet (have it in a config file), for tuner types other than CableCARD, I suspect it will often be needed.
There are some impossibilities in the documentation for the MXF format, such as each keyword group is supposed to use a range of 100 keyword IDs, but some of the keyword groups in my schedulesdirect guide data have over 2700 keywords per group.
There are some inconsistencies between documented JSON fields for the channel tuning info from SD vs. what I actually see in the API responses, plus I have no idea if I can really create tuning info types other than those I am familiar with in the US. I noticed some of the DVBTuningInfo construction functions I could find on my machine are from Nero, and am kind of wondering if DVB tuner manufacturers provide them?
Haven't created a scheduled task yet, or made it so my app can be launched from the console without a UI periodically to import the data, reindex the guide, and update a registry key. You will like what the registry key does --- I have 30 days of guide data from SD, not 12.
Occasionally have required fields missing in the JSON responses. Adding code to handle these cases as they come up.
Still haven't started on anything to map existing scheduled recordings to new seriesIDs/channels when the guide data is changed. I'm wondering how well just matching channel # + time + either series name or at least two episode names would do for this? I think I've figured out how to read what the WMC scheduled recordings are and maybe update them but haven't tried it yet.
Biggest pain point this week is that getting C# XmlSerializer to serialize an object in exactly the way you want is tricky and non-intuitive, and found threads about the documentation being wrong on that from 5 years ago that still hasn't been corrected.
Other things I'm wondering about with it:
The SD data includes station logos, with I think better coverage than MyChannelLogos, but less stylish. To include or not include them?
It also includes not one but usually a dozen or two images per show, with various sizes and other attributes (May be labeled as "banner", "cover art", "scene stills", etc, some may also be labeled as "primary") Also includes images indexed by actor. Currently just picking the one for the show/series closest to ~60K pixels. Going to at least have the size configurable, not sure what other attributes to be able to pick from for the images.
MXF program info includes an "MPAA Rating". SD info includes a list of ratings (plural) per show, each from a different rating agency. Usually MPAA isn't one of them, unless it's a movie. Should there be a mapping from the ratings of the other 20 or so ratings agencies to an MPAA equivalent rating? It seems like some of the XMLTV grabbers do this..
What to do with your existing channels, which may include manually user added channels. I'm thinking of trying to make the channels this creates match the types WMC creates for its own lineups (WMI, CalculatedScanned, etc.) instead of "UserAdded" which is the current default in my guide editor and what it uses if you add a channel through the WMC UI, and removing any channels that look like they were autogenerated by WMC rather than the user.
ClearQAM lineups from SD are extremely rare from what I can tell -- I can tell what channels WMC has found ClearQAM channels on, but generally can't tell what listings they should use, unless you have an InfiniTV to read the channel map from. Not sure if WMC will match the callsigns to my imported listings if you do a channel scan after the MXF import?
One thing I figured out is that XMLTV is far from being a complete specification. As such, things are kludged in to try to work around it, and none of the XMLTV producers includes even close to the complete data set that you can download through the SchedulesDirect JSON API. The specific properties/enumerations that are in the API and the MXF format are obvious parallels BTW -- the Media Center format was **made for** Zap2It data. (Which is still a subset of what SchedulesDirect JSON API has, but XMLTV only supports a very small subset of that, and which subset is inconsistent from one grabber to another since everything about it is kludgy).
Other things:
The MXFImport/MXFExport APIs to load data into WMC are slow and memory intensive even on a fast machine. Just calling the function to make Media Center create a complete MXF of what is currently in your WMC config uses about 6GB of RAM and takes about 15 minutes (or maybe that is because I have both the Rovi lineups the BSEPG lineup in it). The MXF I am creating so far with SD data is more than double the size of the BSEPG one since it has more detail by not going through XMLTV and I haven't even added the schedules or Channels to it yet. Currently has program info (including cast/crew, both long & short descriptions, image URLs), series info, keywords, and keyword groups.
Never found a way to get WMC to create the channels itself from the MXF import, as it does with its own lineups. I am downloading the list of stations/channels for each lineup, and figuring out based on your configured tuners a set of default channel tuning infos for each station. Haven't done the UI for overriding these defaults yet (have it in a config file), for tuner types other than CableCARD, I suspect it will often be needed.
There are some impossibilities in the documentation for the MXF format, such as each keyword group is supposed to use a range of 100 keyword IDs, but some of the keyword groups in my schedulesdirect guide data have over 2700 keywords per group.
There are some inconsistencies between documented JSON fields for the channel tuning info from SD vs. what I actually see in the API responses, plus I have no idea if I can really create tuning info types other than those I am familiar with in the US. I noticed some of the DVBTuningInfo construction functions I could find on my machine are from Nero, and am kind of wondering if DVB tuner manufacturers provide them?
Haven't created a scheduled task yet, or made it so my app can be launched from the console without a UI periodically to import the data, reindex the guide, and update a registry key. You will like what the registry key does --- I have 30 days of guide data from SD, not 12.
Occasionally have required fields missing in the JSON responses. Adding code to handle these cases as they come up.
Still haven't started on anything to map existing scheduled recordings to new seriesIDs/channels when the guide data is changed. I'm wondering how well just matching channel # + time + either series name or at least two episode names would do for this? I think I've figured out how to read what the WMC scheduled recordings are and maybe update them but haven't tried it yet.
Biggest pain point this week is that getting C# XmlSerializer to serialize an object in exactly the way you want is tricky and non-intuitive, and found threads about the documentation being wrong on that from 5 years ago that still hasn't been corrected.
Other things I'm wondering about with it:
The SD data includes station logos, with I think better coverage than MyChannelLogos, but less stylish. To include or not include them?
It also includes not one but usually a dozen or two images per show, with various sizes and other attributes (May be labeled as "banner", "cover art", "scene stills", etc, some may also be labeled as "primary") Also includes images indexed by actor. Currently just picking the one for the show/series closest to ~60K pixels. Going to at least have the size configurable, not sure what other attributes to be able to pick from for the images.
MXF program info includes an "MPAA Rating". SD info includes a list of ratings (plural) per show, each from a different rating agency. Usually MPAA isn't one of them, unless it's a movie. Should there be a mapping from the ratings of the other 20 or so ratings agencies to an MPAA equivalent rating? It seems like some of the XMLTV grabbers do this..
What to do with your existing channels, which may include manually user added channels. I'm thinking of trying to make the channels this creates match the types WMC creates for its own lineups (WMI, CalculatedScanned, etc.) instead of "UserAdded" which is the current default in my guide editor and what it uses if you add a channel through the WMC UI, and removing any channels that look like they were autogenerated by WMC rather than the user.
ClearQAM lineups from SD are extremely rare from what I can tell -- I can tell what channels WMC has found ClearQAM channels on, but generally can't tell what listings they should use, unless you have an InfiniTV to read the channel map from. Not sure if WMC will match the callsigns to my imported listings if you do a channel scan after the MXF import?
-
dublus
- Posts: 29
- Joined: Fri Apr 05, 2013 5:47 pm
- Location:
-
HTPC Specs:
Wow, it sounds like you've got your hands full!
I'm sure there are others lurking here and I'm not the only one wishing for your success.
Please keep us posted on your progress.
I'm sure there are others lurking here and I'm not the only one wishing for your success.
Please keep us posted on your progress.
-
stokd
- Posts: 22
- Joined: Wed Jan 16, 2013 12:32 am
- Location:
-
HTPC Specs:
Appreciate all the work going into this! I have been frustrated with MS issues with the slow EPG navigation and TBA listings for sport events, and thankfully this site exists!
I feel that in the future Windows support of the guide will fade entirely, and this type of App would be necessary for us to keep using the extender hardware we bought.
I feel that in the future Windows support of the guide will fade entirely, and this type of App would be necessary for us to keep using the extender hardware we bought.
-
glugglug
- Posts: 391
- Joined: Thu Jun 09, 2011 1:34 am
- Location:
-
HTPC Specs:
Added the code for this and other projects to GitHub. https://github.com/glugalug/GuideEditin ... ectGrabber
Will add the binaries once the thing is done/usable.
Will add the binaries once the thing is done/usable.
-
glugglug
- Posts: 391
- Joined: Thu Jun 09, 2011 1:34 am
- Location:
-
HTPC Specs:
Anyone want to play "debug the MXF?"
I got an MXF failure with an error something like "found id of unknown type", when importing more than 1 lineup in a single MXF, so I think what it says in the doc about "lineup id must be "l1"", implying there can only be one lineup in the MXF is true.
So now I changed to it merge my SchedulesDirect lineups into a single lineup for the MXF. Getting an error 66% of the way into the import saying "Object reference not set to instance of an object". Doing the import in-process so that it gets caught in the debugger, instead of using the LoadMXF binary, I see that this exception is happening when adding ScheduleEntrys (but can't tell any more than that) (which makes sense as the schedule starts almost exactly 66% of the way into the file).
Been looking at this a lot today and not making much headway.
I uploaded my failing MXF file here: https://www.dropbox.com/s/392dj21gbxkavip/mxf.7z?dl=0 (compressed with 7zip). Anyone have an idea what is wrong with it? Am I missing a program or serviceID that the schedule references somehow?
I got an MXF failure with an error something like "found id of unknown type", when importing more than 1 lineup in a single MXF, so I think what it says in the doc about "lineup id must be "l1"", implying there can only be one lineup in the MXF is true.
So now I changed to it merge my SchedulesDirect lineups into a single lineup for the MXF. Getting an error 66% of the way into the import saying "Object reference not set to instance of an object". Doing the import in-process so that it gets caught in the debugger, instead of using the LoadMXF binary, I see that this exception is happening when adding ScheduleEntrys (but can't tell any more than that) (which makes sense as the schedule starts almost exactly 66% of the way into the file).
Been looking at this a lot today and not making much headway.
I uploaded my failing MXF file here: https://www.dropbox.com/s/392dj21gbxkavip/mxf.7z?dl=0 (compressed with 7zip). Anyone have an idea what is wrong with it? Am I missing a program or serviceID that the schedule references somehow?
-
glugglug
- Posts: 391
- Joined: Thu Jun 09, 2011 1:34 am
- Location:
-
HTPC Specs:
Update: this is bizarre:
Looking at the data in the call stack when the failure occurs, I figured out it is failing while adding the entry for Newsline (Midnight 11/3 UTC).
So I changed my config to not include FIOS1. Gets a bit further. Fails on the same show for FIOS1 HD (501).
If I exclude these 2 channels, the import is successful.
Edit: After re-downloading from SchedulesDirect, I have FIOS1 News Evening Edition in that time slot as well, where it still fails.
I don't see anything unusual about the Program element for that particular show. Although I do see a bug in my output below, the series attribute is supposed to be si<number> not the ID from schedulesdirect, but that is true for everything in my current MXF:
Other interesting things about it:
Rovi just has a generic "FiOS1 TV" entry for that channel across all time slots. I think it may have always said "No data available" in the Zap2It backed media center guide.
Looking at the data in the call stack when the failure occurs, I figured out it is failing while adding the entry for Newsline (Midnight 11/3 UTC).
So I changed my config to not include FIOS1. Gets a bit further. Fails on the same show for FIOS1 HD (501).
If I exclude these 2 channels, the import is successful.
Edit: After re-downloading from SchedulesDirect, I have FIOS1 News Evening Edition in that time slot as well, where it still fails.
I don't see anything unusual about the Program element for that particular show. Although I do see a bug in my output below, the series attribute is supposed to be si<number> not the ID from schedulesdirect, but that is true for everything in my current MXF:
Code: Select all
<Program id="11716" uid="!Program!GSDSH019299680000" title="FiOS1 News Evening Edition" description="Catch up on all of the day's Long Island News and late breaking stories plus continually updated sports scores. Our meteorologists keep an eye on tomorrow's weather so you know what is in store before you step out the door." language="en" year="2014" originalAirDate="2014-05-27T20:00:00.0000000-04:00" keywords="k1,k102" series="si2323" isSeries="true" isNews="true" guideImage="i3974" />
Rovi just has a generic "FiOS1 TV" entry for that channel across all time slots. I think it may have always said "No data available" in the Zap2It backed media center guide.
-
glugglug
- Posts: 391
- Joined: Thu Jun 09, 2011 1:34 am
- Location:
-
HTPC Specs:
Still failing at the same point importing the FIOS1 schedule. Added an option to the app to exclude all channels (and then included JUST FIOS1 to make an MXF for only the one channel, so that it's easier to follow what's in the MXF file and also the import attempt is near instant.
Can anyone tell me what's wrong with this file?
https://www.dropbox.com/s/q9my2nt6i0hlt5n/mxf.xml?dl=0
Can anyone tell me what's wrong with this file?
https://www.dropbox.com/s/q9my2nt6i0hlt5n/mxf.xml?dl=0
-
glugglug
- Posts: 391
- Joined: Thu Jun 09, 2011 1:34 am
- Location:
-
HTPC Specs:
Seeing how the MXF import is all or nothing, and how it gives me basically no information about what is going wrong when it fails, I think this whole approach is going to be terrible from a maintenance/WAF perspective, and I am probably better off trying to add stuff through the undocumented mcepg.dll library directly...
-
dublus
- Posts: 29
- Joined: Fri Apr 05, 2013 5:47 pm
- Location:
-
HTPC Specs:
Agreed! Trying to debug without any feedback is a loser for maintainability, just as you said.
I wish I had some expertise to lend but I'm afraid my programming skills are not up to par (like many E.E.'s, I code as a means to an and).
If it helps at all, at least know there are many of us that are pulling for you.
I wish I had some expertise to lend but I'm afraid my programming skills are not up to par (like many E.E.'s, I code as a means to an and).
If it helps at all, at least know there are many of us that are pulling for you.
-
stuartm
- Posts: 723
- Joined: Mon Nov 05, 2012 8:05 pm
- Location: Longmont, CO
-
HTPC Specs:
I assume the -v option on LoadMXF didn't help any?
-
glugglug
- Posts: 391
- Joined: Thu Jun 09, 2011 1:34 am
- Location:
-
HTPC Specs:
The -v just gives a progress %.
I figured out the order of XML elements in the output was varying, and the keywords were ending up at the end instead of the beginning of the With block, so added explicit OrderAttributes to enforce to element output order, only to get an error something like Microsoft.MediaCenter.ObjectStore.Provider can not be cast to Microsoft.MediaCenter.EPG.Program. I think this is because they show both the provider id and the file local program id being numeric, both starting with 1 in the examples, which would explain why the BSEPG output uses "provider1" instead of the "1" in the MS examples.
Still thinking I need to try updating the schedules/services/programs through mcepg.dll instead of loadmxf because this is too opaque when something goes wrong. And it's all or nothing - if there is one element it doesn't like the entire MXF is rejected, with no option to keep the successfully loaded programs.
I figured out the order of XML elements in the output was varying, and the keywords were ending up at the end instead of the beginning of the With block, so added explicit OrderAttributes to enforce to element output order, only to get an error something like Microsoft.MediaCenter.ObjectStore.Provider can not be cast to Microsoft.MediaCenter.EPG.Program. I think this is because they show both the provider id and the file local program id being numeric, both starting with 1 in the examples, which would explain why the BSEPG output uses "provider1" instead of the "1" in the MS examples.
Still thinking I need to try updating the schedules/services/programs through mcepg.dll instead of loadmxf because this is too opaque when something goes wrong. And it's all or nothing - if there is one element it doesn't like the entire MXF is rejected, with no option to keep the successfully loaded programs.
-
STC
- Posts: 6808
- Joined: Mon Jun 06, 2011 4:58 pm
- Location:
-
HTPC Specs:
Would you mind posting your bat files / new tweaks. I am going to fire up new test MC enviros for both your and glugglugs zap2it solutions.audinutt wrote:The zap 2 it epg has been working really well with this final tweak.
By the Community, for the Community. 100% Commercial Free.
Want decent guide data back? Check out EPG123
Want decent guide data back? Check out EPG123
-
audinutt
- Posts: 114
- Joined: Thu May 14, 2015 2:43 am
- Location:
-
HTPC Specs:
Will do later tonight