Detailed EPG Guide Data in Australia (via xmltv.net)

Global guide related discussion. All Windows versions.
Post Reply
th3luk3st3r

Posts: 24
Joined: Fri Jun 12, 2020 11:23 am
Location:

HTPC Specs: Show details

Detailed EPG Guide Data in Australia (via xmltv.net)

#1

Post by th3luk3st3r » Thu Mar 18, 2021 11:46 pm

Hi fellow WMC enthusiasts! :D

I live in Australia where EPG123 can't be of service due to the unavailability of SchedulesDirect guide data, leaving me at either the mercy of IceTV or the OTA EPG. While WMC can pull in guide information from the DVB-T signal over the air, the experience is sub-optimal, and the only data provided is a basic title, basic description, category, (and of course) start and end times of the programme. This allows for basic recording but the double-ups and lack of information degrade the experience significantly.

Over the last few months, I've been working on a tool to convert XMLTV files from xmltv.net (a more detailed EPG source that is intended for use with Plex DVR) to MXF files that WMC can import and use. Initially I was hoping to use the already available xmltv-to-mxf tool, but found that these specific XMLTV files were somehow incompatible with the XMLTV spec and that xmltv-to-mxf kept crashing because of it; the next best option to build my own tool ... so I did! ;)

Half of the tool sits in Azure, where (every 6 hours) it loads in the latest guide data from xmltv.net and converts the XMLTV to MXF. The other half runs as a Windows Service, where (also every 6 hours) it downloads the created MXF file (and any required channel logos to speed up the WMC Guide), and imports the data into WMC using LoadMXF.

This results in a much more rich EPG experience; HD channels and programmes are marked as such; there are guide images for the majority of programmes; seasons, series, and episode titles are recognised as they should be; and extra program categories allow for more reliable program discovery. All the while, the MXF UIDs are persisted into a DB in Azure allowing for continuity between guide imports. It's *almost* as good as IceTV...

It's currently in testing but if anyone else in Australia lives in a region covered by xmltv.net and would like to give it a go, feel free to reach out. :)

User avatar
garyan2

Posts: 7438
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#2

Post by garyan2 » Fri Mar 19, 2021 12:24 am

If you want me to checkout your generated MXF file, let me know. I've learned a thing or 2 along the way here. I don't know what the xmltv.net file looks like as far as any unique programme/series info it might have, but you might not have to maintain any UIDs between updates.
- Gary
Keeping WMC alive beyond January 2020. https://garyan2.github.io

th3luk3st3r

Posts: 24
Joined: Fri Jun 12, 2020 11:23 am
Location:

HTPC Specs: Show details

#3

Post by th3luk3st3r » Sun Mar 21, 2021 6:51 am

Wow, I would greatly appreciate that @garyan2! As I'm only new to the MXF guide data party, I'd appreciate any help or advice :D

My first shot at importing the guide data didn't persist the UIDs and (whether coincidence or not) I seemed to be having issues with programmes being recorded under the wrong data and Recorded TV displaying the wrong programme titles; though from the sounds of it I was obviously missing something with my implementation :) If I don't have to persist any UIDs then that makes life 1000% easier ;)

If you want to have a look at a generated MXF file (I would be very grateful!), http://mxfservice.azurewebsites.net/latest will automatically download the latest generated version of the guide. :)

User avatar
garyan2

Posts: 7438
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#4

Post by garyan2 » Wed Mar 24, 2021 4:05 am

Sorry, I went down the rabbit hole on this and actually created my own program to see what yourtv.com.au provides. Unfortunately, your xmltv source doesn't take advantage of some of the flags yourtv provides such as series IDs and program IDs, new/repeat/premiere flags, and they do some odd things concerning the xmltv original-air-date, previously-shown, and premiere parts.

General Statement
For any string attribute that is empty, you don't need to serialize it to the xml. Same for any boolean that is false and integer that is 0. This will significantly reduce the size of your output file.

SeriesInfos

Code: Select all

<SeriesInfo id="si1" uid="!Series!2" title="The Late Show with Stephen Colbert" shortTitle="The Late Show with Stephen Colbert" />
Rather than use a number for the uid of the series, a quick way would be to simply use the title (same way the person is done in the file), but I would suggest using the hash of the title (1740122544). Overall cleaner look. Also, if the title and shortTitle are the same, then just use the title and leave the shortTitle empty.

Seasons

Code: Select all

<Season id="sn1" uid="!Season!2" series="si1" title="The Late Show with Stephen Colbert: Season 6" year="0" />
For the season, I would suggest using the hash of the series title like above and just append the season number like "1740122544_6".

Programs

Code: Select all

<Program id="1" uid="!Program!7176" title="The Late Show with Stephen Colbert" description="Join Stephen Colbert as he welcomes guests Robert Downey Jr and Walter Isaacson, with a musical performance by Sebastian Yatra and Guyanaa." shortDescription="Join Stephen Colbert as he welcomes guests Robert Downey Jr and Walter Isaacson, with a musical performance by Sebastian Yatra and Guyanaa." episodeTitle="Robert Downey Jr., And Walter Isaacson, Sebastian Yatra And Guyanaa" year="0" seasonNumber="6" episodeNumber="142" keywords="k300" halfStars="0" mpaaRating="0" series="si1" season="sn1" isSeries="true" isMovie="false" isShort="false" isMiniseries="false" isPaidProgramming="false" isKids="false" isSports="false" isSpecial="false" isNews="false" isReality="false" guideImage="i62" />
More later, but same note for the description and shortDescription attributes don't both need to be filled. Considering your xmltv source, there is only going to be one description so don't worry about the shortDescription at all. The uid won't be too complicated, but how we treat each program is the challenging part considering your xmltv source.

ScheduleEntries

Code: Select all

<ScheduleEntry program="1" startTime="2021-03-23T13:00:00" duration="3600" isCC="false" isHdtv="true" isPremiere="false" isFinale="false" isLive="false" isLiveSports="false" isTape="false" isDelay="false" isSubtitled="false" audioFormat="0" part="0" parts="0" />
Nothing to say here, this is fine with the same notes for serialization.

Lineups/channels

Code: Select all

<Channel uid="!Channel!Adelaide!1" lineup="l1" service="s1" number="1" matchName="10 HD" />
This works. What you have in the matchName doesn't do anything for you so I would just leave it empty. The correct usage for DVBT is "DVBT:onid:tsid:sid" so that it could automatically map to your tuner channels (first time only) upon import.
- Gary
Keeping WMC alive beyond January 2020. https://garyan2.github.io

User avatar
garyan2

Posts: 7438
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#5

Post by garyan2 » Wed Mar 24, 2021 4:57 am

Now the problem with your xmltv source for the programs is they ignore the provided series and program ids from yourtv, and do some strange things to make the programs unique.

yourtv provides 9 categories for program types.
  1. Television movie - Movie that premieres on television (not video or cinema).
  2. Cinema movie - Movie that premiered in cinema (not television or video).
  3. Mini series - A short series with a continuing storyline. Instalments are usually of movie length, with part numbers.
  4. Series no episodes - Program that airs frequently and continuously, but does not have a continuing storyline and/or does not have specific episode titles.
  5. Series with episodes - Episodic series with episode titles. May or may not have continuing storyline. Runs more than 6 episodes.
  6. Serial - Fictional series with continuing storyline - serials and soaps. Runs more than 12 episodes.
  7. Video movie - Movie that premieres on video (not television or cinema).
  8. Limited series - Short series that may have continuing storyline (episodes) or not. Runs less than 12 episodes. Usually documentaries.
  9. Special - Program with only one instalment. A one-off show.
Your xmltv provider does the following depending on the category.
For the movies (1,2,7) they don't do anything special but unfortunately do not provide the year of release.
For the mini series (3), it is always season 1 and will include the episode number if available.
For the series with no episodes (4), they set the premiere flag in the xmltv, provide an original-air-date of the start time of the program, and use the season and episodes numbers if available... if not available, they use the year for the season number and the month&day for the episode number.
For the specials (9), the season number is always the year, and the episode number is always the month&day.
For the rest (5,6,8), the season number and episode number follow the same rules as #4.

The one thing they did get right for us is if yourtv flagged a program as repeat, then they would provide the previously-shown element.

So, all the above is just to lay out the understanding that we absolutely have no idea what the original airdate of a program is, so don't populate that. With no OAD, everything will be treated as new. I believe you could provide an OAD of "1970-01-01", as an example, for any programme that has the previously-shown element. It would then be shown as a repeat in the guide.

So I'm going to give a couple options for the program uid.
1. Create a hash for the program to include the title, episodetitle, and description. I did something similar for the hdhomerun slice guide but I had more information available to me to make it more robust. Below can be an example.

Code: Select all

public override int GetHashCode()
{
    var ret = (Title != null) ? Title.GetHashCode() : 0;
    ret = (ret * 397) ^ (EpisodeNumber != null ? EpisodeNumber.GetHashCode() : 0);
    ret = (ret * 397) ^ (EpisodeTitle != null ? EpisodeTitle.GetHashCode() : 0);
    ret = (ret * 397) ^ (Synopsis != null ? Synopsis.GetHashCode() : 0);
    ret = (ret * 397) ^ (Team1 != null ? Team1.GetHashCode() : 0);
    ret = (ret * 397) ^ (Team2 != null ? Team2.GetHashCode() : 0);
    ret = (ret * 397) ^ SeriesId.GetHashCode();
    ret &= 0x7fffffff;

    return ret;
}
2. Just use the hash code of the title and add the season/episode number (ex. 1740122544_S6E142). Because of what your xmltv source does with the season and episode numbers above, doing this will force you to have the same program repeated multiple times in the mxf file for things like daily news programs.
- Gary
Keeping WMC alive beyond January 2020. https://garyan2.github.io

th3luk3st3r

Posts: 24
Joined: Fri Jun 12, 2020 11:23 am
Location:

HTPC Specs: Show details

#6

Post by th3luk3st3r » Tue Apr 13, 2021 1:12 am

I apologise for the slow response - leading a busy life is my excuse :D

Thanks for the suggestions, Gary! I have implemented pretty much all of them, but took things a step further and built out the application to pull the data directly from yourtv.com.au and the other DTV radio EPG sources, rather than using the xmltv.net files. This has eliminated the issues with the xmltv.net XMLTV file not containing the yourtv.com.au IDs and program types; so I've used the IDs wherever possible, and used a hash where no ID is provided.

garyan2 wrote: Wed Mar 24, 2021 4:05 am This works. What you have in the matchName doesn't do anything for you so I would just leave it empty. The correct usage for DVBT is "DVBT:onid:tsid:sid" so that it could automatically map to your tuner channels (first time only) upon import.
I gave this a shot using the DVB triplet that can be pulled from FreeView AU for the major TV channels in the area; however, this does not seem to work with automatic channel mapping after running MediaCenterRecoveryTask (and then reconfiguring TV), or even with a fresh (never used) copy of WMC on a fresh installation of Windows 7. My only though is that perhaps the triplets provided by FV AU are incorrect? Oddly enough though, I have found that if MatchName exactly matches the OTA channel's name then WMC does automatically map the channel. Very intriguing... :think:

User avatar
garyan2

Posts: 7438
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#7

Post by garyan2 » Tue Apr 13, 2021 2:06 am

Yah, I took a peek last week at where you were and noticed you went to the source. I didn't look at the radio stuff... is that where your programs without IDs are coming from?

Weird about the matchName. I do know that if the name matches the callsign of the channel, it should work, but I'm surprised the triplet didn't. One of the strange things as well is that I noticed the triplet (in my case doublet for OTA) would only automatically match on a clean channel. Meaning the merged channel could never have had guide listings assigned to it before... a clean WMC setup.

The only thing I would add from my peak at your mxf file is you don't need to have an original airdate for movies. You just need the year.
- Gary
Keeping WMC alive beyond January 2020. https://garyan2.github.io

rkulagow

Posts: 246
Joined: Sun Jul 19, 2015 1:04 am
Location: Schedules Direct

HTPC Specs: Show details

#8

Post by rkulagow » Sun Jun 13, 2021 2:45 am

th3luk3st3r wrote: Thu Mar 18, 2021 11:46 pm I live in Australia where EPG123 can't be of service due to the unavailability of SchedulesDirect guide data, leaving me at either the mercy of IceTV or the OTA EPG. While WMC can pull in guide information from the DVB-T signal over the air, the experience is sub-optimal, and the only data provided is a basic title, basic description, category, (and of course) start and end times of the programme. This allows for basic recording but the double-ups and lack of information degrade the experience significantly.
(I work for Schedules Direct)

We do have Australian Guide Data now; does Australia not appear as a country?

th3luk3st3r

Posts: 24
Joined: Fri Jun 12, 2020 11:23 am
Location:

HTPC Specs: Show details

#9

Post by th3luk3st3r » Sun Jun 13, 2021 3:03 am

rkulagow wrote: Sun Jun 13, 2021 2:45 am
th3luk3st3r wrote: Thu Mar 18, 2021 11:46 pm I live in Australia where EPG123 can't be of service due to the unavailability of SchedulesDirect guide data, leaving me at either the mercy of IceTV or the OTA EPG. While WMC can pull in guide information from the DVB-T signal over the air, the experience is sub-optimal, and the only data provided is a basic title, basic description, category, (and of course) start and end times of the programme. This allows for basic recording but the double-ups and lack of information degrade the experience significantly.
(I work for Schedules Direct)

We do have Australian Guide Data now; does Australia not appear as a country?
I had a look on the data regions page here https://www.schedulesdirect.org/regions and it doesn't mention Australia, so I took that to mean that there wasn't data available. If there is, that'd be the best news I've had all week!

User avatar
garyan2

Posts: 7438
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#10

Post by garyan2 » Sun Jun 13, 2021 4:05 am

Australia is new to the regional lineup options from SD and do show up in epg123 for selection. I think it was not available back in April.
australia.PNG
australia.PNG (10.48 KiB) Viewed 9076 times
- Gary
Keeping WMC alive beyond January 2020. https://garyan2.github.io

saxnix

Posts: 53
Joined: Thu Mar 01, 2012 10:37 pm
Location:

HTPC Specs: Show details

#11

Post by saxnix » Tue Sep 26, 2023 8:58 am

Not sure if this is the best place for this question, but who do we provide feedback to about the data being incorrect?

The guide data for the most part is correct (and it's excellent to have), but quite a few first-run programs (mainly USA prime-time shows) are incorrectly flagged as a Repeat. This then requires changing the settings for the affected shows to record New & Rerun episodes. Doing so will then record episodes that are reruns and the 2nd time the same episode plays on the same evening (but later). Any advice here? TIA

UPDATE: addded the following
New (first airing) episodes for some shows are showing the "Repeat" flag (this mostly happens with US programs), and some Reruns aren't showing the rerun tag when they should. It has been this way since I joined the service, I just hadn't got around to reporting it.

For example, each day The Block is on Channel 9 (first run) at 7:30pm, and is Rerun the following day at noon. The noon airing does not have the Repeat flag.

The Project, Channel 10 (first run) at 6:30pm is often repeated around 11:30pm the same day. The later timeslot does not contain the Repeat flag.

Chicago Med, Channel 9 (S6:E12) is showing the Repeat flag but I believe this is the first run of this season.

Chicago Fire, Channel 7 (S11:E12) is showing the Repeat flag but I believe this is the first run of this season.

My Kitchen Rules, Channel 7 (S13:E11) first run and rerun the following day around noon, the noon showing does not show the Repeat flag.

Shark Tank, Channel 10 (S5:E5) first run at night, rerun the following day around 2pm. Rerun not showing the Repeat flag.

These are just some examples for this week.

User avatar
garyan2

Posts: 7438
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#12

Post by garyan2 » Tue Sep 26, 2023 9:13 pm

I'm assuming you are using EPG123, rather than xmltv.net like the original post.

For the NEW airings showing as REPEAT, do you have the option to allow a NEW flag to override the original air date enabled?

The rest is just the way WMC works. If an airing is within 3 days of the original air date, it is considered new. Outside of 3 days, it is considered repeat.

Sent from my SM-G781U1 using Tapatalk

- Gary
Keeping WMC alive beyond January 2020. https://garyan2.github.io

Space

Posts: 2838
Joined: Sun Jun 02, 2013 9:44 pm
Location:

HTPC Specs: Show details

#13

Post by Space » Tue Sep 26, 2023 11:14 pm

Some of your issues seem to be related to a misunderstanding of how WMC works.

WMC does not use the "new" flag from the guide source, it instead relies on the OAD (Original Air Date) of the episode.

If an airing of an episode is within 7 days (not 3) of the OAD then it is considered "new", otherwise it is marked as a "Repeat". This is why you will see multiple airings of the same episode without the "Repeat" indicator, as they are all considered "new" as far as WMC is concerned. Note that this does not cause an issue with repeated recordings of the same episode, as I will explain below.

I assume you are in Australia, since you responded to this thread. Some of the shows you are complaining about originally aired in another country, so I assume that the OAD is the date they aired in that country, as opposed to your own country.

If this is the case, this is why the episodes are considered "Repeats", as the airing date is not within 7 days of the OAD.

Note that for this situation, I advise you to set your Series to "New & Rerun" and to record from a specific channel, as this will record ALL unique episodes of a show that have not already been recorded by your Series on that channel. A Series should not record the same episode twice (even if set to "New & Rerun"), so your fear that the Series would record the original airing and the next day airing (or any other airing of that same episode) are unfounded (although there are some bugs in WMC that might occasionally result in the same episode being recorded more than once, this is not the norm).

If you would rather use the "New only" setting for your Series, then you can try enabling the option suggested by Gary in the previous post. What this setting does is set the OAD for the episode to be the same as the date of the airing of that episode that has the "new" flag set in the guide source data. However using this option completely relies on the "new" flag being correct in the source data, which it often is not, particularly for airings of foreign programs (at least in my experience). This is why I recommend using the "New & Rerun" and specific channel settings instead, unless there is a good reason not to (such as many repeats of the show from previous seasons airing on the same channel).

If the "New & Rerun" setting results in WMC scheduling some older episode to record that you do not want recorded, you can select them and choose "Do not record" and they will be added to the list of episode to not record for that Series. If there are a LOT of older episodes like this and you don't want to keep having to manually tell WMC to not record them every week, then you may want to use the "New only" option as described above instead of "New & Rerun", with the caveat that you will then be relying on the "new" flag to be set correctly in the source guide data.

Post Reply