Introducing EPG123

[garyan2]

I'm glad to see that the cache will finally be a single file. That should substantially improve the disk thrashing. Are there any instructions or concerns about upgrading? What happens to the tens of thousands of old cache files on an upgraded system and what do you suggest as memory requirements in order to see any performance gain?

Nothing special to do or any concerns. It's just the first time you run the new version, it will take a bit longer to create the new cache file and delete all the old ones.

Memory requirements will certainly be dependent on the number of stations and days you are downloading. With 680 stations and 15 days, with no cached files and creating an XMLTV, memory usage got up to 1650MB for my tests. The same number of stations and days, with all data cached and no XMLTV, required 870MB.

[glorp]

Yup got rid of ~60,000 cache files for one ~140K JSON with the first run this morning. That should speed things up going forward and reduce disk thrashing substantially. Nice job. Didn't really pay attention to memory use since it's being run on the headless server but that has 16GB available so shouldn't be an issue.

[glorp]

My task run times have fallen from 12-15 minutes to under 2.

[garyan2]

Yah, I really wish I had done this a long time ago. I had always imagined I'd have to learn to use SQL/SQLite or something, but once I sat down and really looked at it, my solution was simple and the efforts paid out in spades. It's also nice to have a single file that I can search through to find info for troubleshooting if need be.

I'm now researching a little into how to compress the file, but I think it might take too much memory (serializing the data to a memory stream, to then send to a compression stream and to a file stream), but I could probably get the cache file to 15% of its' current size with normal compression.

[garyan2]

All right, I've got compression going on. The table below says it all. This is with maximum compression. I'm going to try and benchmark on some slow systems to see if there are any side effects.

cacheImprovements.PNG (6.85 KiB) Viewed 4666 times

[Space]

Personally I think it's fine the way it is. Not sure disk space is a huge concern for most people. The only reason I mentioned the disk space was due to the slack space and the efficiency aspect. The actual amount of space it took up was of little concern to me.

Plus I also like the idea that I can look through the cache file, although I probably will never do it again.

I'd rather have it run fast then save some disk space, but if it is a negligible increase in execution time then I guess it's fine.

Note that ages ago I did some tests with different compression algorithms for text (not binary) data and the ones that are better at text compression can be orders of magnitude better then generic compression algorithms that are designed for binary data.

Although, like I said, that was years ago, so compression may have improved since then.

EDIT:
Just looked back at my old test results and the best algorithm for compressing text at that time was PPMd with 7zip. Still looks like this is one of the best ones as far as speed and compression.

https://www.quora.com/What-is-the-best- ... -algorithm

Here's one of the notes I took:
gzip 16.697% of original size
7zip 2.727% of original size

I believe that gzip was using Lempel-Ziv and 7zip was using PPMd.

[IT Troll]

Oh interesting, I’ll have to get upgraded. I am guessing those running from SSD will not see much difference though?

[garyan2]

Not quite as much, but I did see a difference. For an update run that was typically around 22s, I am now around 16s with the .20 version. This is on a circa '16 Mushkin SSD.

[npradeepkumar]

I have a question. Will this work with Windows 10, if I buy the media center pack? I had this up and running and was working fine, but recently my machine crashed and I had to re-install the OS and start over. Since Windows 7 is officially not supported anymore, it has been throwing hurdles every step of the way with lots of other software. So, I am debating if I should bite the bullet and switch to Windows 10.

I don't know if there are other issues or gotchas that I need to be aware of (if I am switching to Windows 10).

Thanks

[stuartm]

Be aware that Windows 10 does not support WMC, it will only work with a fair bit of hacking on your part. Also AIUI only OTA works, I don't think cable card will work with the win10 hacks (and definitely not any protected content). There is a whole subforum about WMC and win10 here viewforum.php?f=97. That said, if you go that route EPG123 does work with the hacked in version of WMC for win10.

[npradeepkumar]

Thanks for the warnings. If protected content and premium channels like ESPN etc. do not work, then it is a no go for me. I guess, I have to stay on Windows 7 and make it work.

I have been looking at other options for whole home media solutions. I briefly looked at Plex, Silicondust (hd homerun). There seems to be no good option out there that can do all the things that WMC can do on Windows 7. What else am I missing? Is Direct Tv a better solution or is there any other solution that is better or at least equivalent with all the features that WMC has (Connectivity in multiple rooms, ability to play all channels including premium, DVR functionality etc.)

Apologize if this is not the right place to ask this question.

[sptrainman]

Hi,

My guide has not updated for the last two nights now. The problem seems to be due to missing channels in the data coming from Schedules Direct. Last night, the mxf imported, but the file is 72 hours old. So I'm not sure what that means.

I sent the trace log to you via email.

Thanks,
Jerry

[garyan2]

Hi,

My guide has not updated for the last two nights now. The problem seems to be due to missing channels in the data coming from Schedules Direct. Last night, the mxf imported, but the file is 72 hours old. So I'm not sure what that means.

I sent the trace log to you via email.

Thanks,
Jerry

Jerry, I haven't received your trace.log file. Please send to support@garyan2.net.

EPG123 has a safety check whereas if the available number of stations being downloaded is less than 95% of the number of stations that were available when you last saved the configuration file, it will abort the update. This avoids a massive loss of channels in your guide that may or may not be real.

[garyan2]

Hi,

My guide has not updated for the last two nights now. The problem seems to be due to missing channels in the data coming from Schedules Direct. Last night, the mxf imported, but the file is 72 hours old. So I'm not sure what that means.

I sent the trace log to you via email.

Thanks,
Jerry

Jerry, I haven't received your trace.log file. Please send to support@garyan2.net.

EPG123 has a safety check whereas if the available number of stations being downloaded is less than 95% of the number of stations that were available when you last saved the configuration file, it will abort the update. This avoids a massive loss of channels in your guide that may or may not be real.

Got it. When you last saved your configuration file, you had 674 stations selected for download. On 8/21 you were downloading 687 stations meaning there were "new" stations added since then. The problem is on 8/22 that number of stations available for download was decreased to 633 which is 93.9% of the original 674. So it is 41 stations less than what were available when you last saved your configuration file, and 54 less than what was available on 8/21.

I'm not going to be able to tell what disappeared and/or added since you last saved your configuration file, but I suggest you open the configuration GUI and inspect to see if any stations you care about are missing. If you are okay with what is listed, just click the [Save & Execute] and everything will complete. Performing a save re-baselines the number of stations you are expecting to download to 633 instead of 674 and it will pass the safety check.

[NaivekGB]

Since last week, I am noticing the same error reported in this thread, did a upgrade to recent build epg123Setup_v1.6.3.0 and it keep failing to update the guide. Could you please look at the logs and let me know if there is a way to fix this?

Thanks

Code: Select all

[3/25/2021 12:00:01 AM] ===============================================================================
[3/25/2021 12:00:01 AM]  Beginning epg123 update execution. version 1.6.3.0
[3/25/2021 12:00:01 AM] ===============================================================================
[3/25/2021 12:00:01 AM] [ INFO] Beginning EPG123 update execution. 2021-03-25 07:00:01Z
[3/25/2021 12:00:01 AM] [ INFO] DaysToDownload: 14 , TheTVDBNumbers : True , PrefixEpisodeTitle: False , PrefixEpisodeDescription : False , AppendEpisodeDesc: False , OADOverride : True , TMDbCoverArt: True , IncludeSDLogos : True , AutoAddNew: True , CreateXmltv: False , ModernMediaUiPlusSupport: False
[3/25/2021 12:00:02 AM] [ INFO] Token request successful. serverID: 20141201.web.1
[3/25/2021 12:00:02 AM] [ INFO] Status request successful. account expires: 2021-07-22T00:27:01Z , lineups: 1/4 , lastDataUpdate: 2021-03-24T22:32:16Z
[3/25/2021 12:00:02 AM] [ INFO] system status: Online , message: No known issues.
[3/25/2021 12:00:04 AM] [ INFO] Successfully retrieved TMDb configurations.
[3/25/2021 12:00:04 AM] [ INFO] Successfully requested listing of client lineups from Schedules Direct.
[3/25/2021 12:00:04 AM] Entering buildLineupServices() for 1 lineups.
[3/25/2021 12:00:05 AM] [ INFO] Successfully retrieved the station mapping for lineup USA-WA62449-X.
[3/25/2021 12:00:07 AM] Exiting buildLineupServices(). SUCCESS.
[3/25/2021 12:00:07 AM] Entering getAllScheduleEntryMd5s() for 14 days on 616 stations.
[3/25/2021 12:02:10 AM] [ INFO] Successfully retrieved Md5s for 357 station's daily schedules. (0:00:02:02.9622244 /   532.248 KB)
[3/25/2021 12:07:10 AM] [ INFO] SD API WebException Thrown. Message: The operation has timed out , Status: Timeout . Trying again.
[3/25/2021 12:12:10 AM] [ INFO] SD API WebException Thrown. Message: The operation has timed out , Status: Timeout . Trying again.
[3/25/2021 12:12:10 AM] [ERROR] Failed to complete request. Exiting
[3/25/2021 12:12:10 AM] [ERROR] Did not receive a response from Schedules Direct for 357 station's daily schedules. (0:00:10:00.0234984)
[3/25/2021 12:12:10 AM] [ERROR] Problem occurred during getMd5ScheduleEntries(). Exiting.
[3/25/2021 12:12:10 AM] [ERROR] Failed to create MXF file. Exiting.
[3/25/2021 12:12:10 AM] [ INFO] epg123 update execution time was 00:12:09.2049562.
[3/25/2021 12:12:11 AM] ===============================================================================
[3/25/2021 12:12:11 AM]  Beginning epg123 client execution. version 1.6.3.0
[3/25/2021 12:12:11 AM] ===============================================================================
[3/25/2021 12:12:11 AM] [ INFO] Beginning epg123 client execution. 2021-03-25 07:12:11Z
[3/25/2021 12:12:11 AM] [ INFO] Import: True , Match: True , NoLogo: False , Force: False , ShowProgress: False , NoGC: False , NoVerify: False , Verbose: False
[3/25/2021 12:12:12 AM] Entering ImportMxfFile() for file "c:\programdata\garyan2\epg123\output\epg123.mxf".
[3/25/2021 12:14:02 AM] [ INFO] Successfully imported .mxf file into Media Center database. Exit code: 0
[3/25/2021 12:14:02 AM] Exiting ImportMxfFile(). SUCCESS.
[3/25/2021 12:14:02 AM] Entering VerifyLoad()
[3/25/2021 12:15:21 AM] [ INFO] Checked 197050 entries and corrected 0 of them.
[3/25/2021 12:15:21 AM] Exiting VerifyLoad()
[3/25/2021 12:15:25 AM] [ INFO] Completed the automatch of lineup stations to tuner channels.
[3/25/2021 12:15:26 AM] [ INFO] Completed lineup refresh.
[3/25/2021 12:15:26 AM] Entering RunWmcIndexTask(PvrScheduleTask)
[3/25/2021 12:15:26 AM] [ INFO] SUCCESS: Attempted to run the scheduled task "Microsoft\Windows\Media Center\PvrScheduleTask".
[3/25/2021 12:15:26 AM] [ INFO] Successfully started the PvrScheduleTask task. Exit code: 0
[3/25/2021 12:15:26 AM] Exiting RunWmcIndexTask(PvrScheduleTask). SUCCESS.
[3/25/2021 12:15:26 AM] Entering RunWmcIndexTask(ReindexSearchRoot)
[3/25/2021 12:15:26 AM] [ INFO] SUCCESS: Attempted to run the scheduled task "Microsoft\Windows\Media Center\ReindexSearchRoot".
[3/25/2021 12:15:26 AM] [ INFO] Successfully started the ReindexSearchRoot task. Exit code: 0
[3/25/2021 12:15:26 AM] Exiting RunWmcIndexTask(ReindexSearchRoot). SUCCESS.
[3/25/2021 12:15:28 AM] [ INFO] MXF file was created on 3/21/2021 3:05:43 PM
[3/25/2021 12:15:28 AM] [ERROR] The MXF file imported is 81.16 hours old.
[3/25/2021 12:15:30 AM] [ INFO] Completed EPG123 client execution.
[3/25/2021 12:15:30 AM] [ INFO] EPG123 client execution time was 00:03:19.2546155.
[3/26/2021 12:00:01 AM] ===============================================================================
[3/26/2021 12:00:01 AM]  Beginning epg123 update execution. version 1.6.3.0
[3/26/2021 12:00:01 AM] ===============================================================================
[3/26/2021 12:00:01 AM] [ INFO] Beginning EPG123 update execution. 2021-03-26 07:00:01Z
[3/26/2021 12:00:01 AM] [ INFO] DaysToDownload: 14 , TheTVDBNumbers : True , PrefixEpisodeTitle: False , PrefixEpisodeDescription : False , AppendEpisodeDesc: False , OADOverride : True , TMDbCoverArt: True , IncludeSDLogos : True , AutoAddNew: True , CreateXmltv: False , ModernMediaUiPlusSupport: False
[3/26/2021 12:00:02 AM] [ INFO] Token request successful. serverID: 20141201.web.1
[3/26/2021 12:00:03 AM] [ INFO] Status request successful. account expires: 2021-07-22T00:27:01Z , lineups: 1/4 , lastDataUpdate: 2021-03-26T02:35:03Z
[3/26/2021 12:00:03 AM] [ INFO] system status: Online , message: No known issues.
[3/26/2021 12:00:04 AM] [ INFO] Successfully retrieved TMDb configurations.
[3/26/2021 12:00:04 AM] [ INFO] Successfully requested listing of client lineups from Schedules Direct.
[3/26/2021 12:00:04 AM] Entering buildLineupServices() for 1 lineups.
[3/26/2021 12:00:05 AM] [ INFO] Successfully retrieved the station mapping for lineup USA-WA62449-X.
[3/26/2021 12:00:07 AM] [ INFO] Kicking off background worker to download and process 1 station logos.
[3/26/2021 12:00:07 AM] Exiting buildLineupServices(). SUCCESS.
[3/26/2021 12:00:07 AM] Entering getAllScheduleEntryMd5s() for 14 days on 616 stations.
[3/26/2021 12:02:06 AM] [ INFO] Successfully retrieved Md5s for 357 station's daily schedules. (0:00:01:58.1989145 /   532.255 KB)
[3/26/2021 12:07:06 AM] [ INFO] SD API WebException Thrown. Message: The operation has timed out , Status: Timeout . Trying again.
[3/26/2021 12:07:07 AM] [ INFO] SD API WebException Thrown. Message: The remote server returned an error: (502) Bad Gateway. , Status: ProtocolError
[3/26/2021 12:07:07 AM] [ERROR] Failed to complete request. Exiting
[3/26/2021 12:07:07 AM] [ERROR] Did not receive a response from Schedules Direct for 357 station's daily schedules. (0:00:05:01.4809963)
[3/26/2021 12:07:07 AM] [ERROR] Problem occurred during getMd5ScheduleEntries(). Exiting.
[3/26/2021 12:07:08 AM] [ERROR] Failed to create MXF file. Exiting.
[3/26/2021 12:07:08 AM] [ INFO] epg123 update execution time was 00:07:06.8550902.
[3/26/2021 12:07:08 AM] ===============================================================================
[3/26/2021 12:07:08 AM]  Beginning epg123 client execution. version 1.6.3.0
[3/26/2021 12:07:08 AM] ===============================================================================
[3/26/2021 12:07:08 AM] [ INFO] Beginning epg123 client execution. 2021-03-26 07:07:08Z
[3/26/2021 12:07:08 AM] [ INFO] Import: True , Match: True , NoLogo: False , Force: False , ShowProgress: False , NoGC: False , NoVerify: False , Verbose: False
[3/26/2021 12:07:09 AM] Entering PerformGarbageCleanup().
[3/26/2021 12:49:24 AM] [ INFO] Successfully completed garbage cleanup. Exit code: 0
[3/26/2021 12:49:24 AM] Exiting PerformGarbageCleanup(). SUCCESS.
[3/26/2021 12:49:26 AM] Entering ImportMxfFile() for file "c:\programdata\garyan2\epg123\output\epg123.mxf".
[3/26/2021 12:51:07 AM] [ INFO] Successfully imported .mxf file into Media Center database. Exit code: 0
[3/26/2021 12:51:07 AM] Exiting ImportMxfFile(). SUCCESS.
[3/26/2021 12:51:07 AM] Entering VerifyLoad()
[3/26/2021 12:52:04 AM] [ INFO] Checked 197050 entries and corrected 17 of them.
[3/26/2021 12:52:04 AM] Exiting VerifyLoad()
[3/26/2021 12:52:08 AM] [ INFO] Completed the automatch of lineup stations to tuner channels.
[3/26/2021 12:52:09 AM] [ INFO] Completed lineup refresh.
[3/26/2021 12:52:09 AM] Entering RunWmcIndexTask(PvrScheduleTask)
[3/26/2021 12:52:09 AM] [ INFO] SUCCESS: Attempted to run the scheduled task "Microsoft\Windows\Media Center\PvrScheduleTask".
[3/26/2021 12:52:09 AM] [ INFO] Successfully started the PvrScheduleTask task. Exit code: 0
[3/26/2021 12:52:09 AM] Exiting RunWmcIndexTask(PvrScheduleTask). SUCCESS.
[3/26/2021 12:52:09 AM] Entering RunWmcIndexTask(ReindexSearchRoot)
[3/26/2021 12:52:09 AM] [ INFO] SUCCESS: Attempted to run the scheduled task "Microsoft\Windows\Media Center\ReindexSearchRoot".
[3/26/2021 12:52:09 AM] [ INFO] Successfully started the ReindexSearchRoot task. Exit code: 0
[3/26/2021 12:52:09 AM] Exiting RunWmcIndexTask(ReindexSearchRoot). SUCCESS.
[3/26/2021 12:52:09 AM] [ INFO] MXF file was created on 3/21/2021 3:05:43 PM
[3/26/2021 12:52:09 AM] [ERROR] The MXF file imported is 105.77 hours old.
[3/26/2021 12:52:11 AM] [ INFO] Completed EPG123 client execution.
[3/26/2021 12:52:11 AM] [ INFO] EPG123 client execution time was 00:45:03.2917421.

[Gripweed]

Try changing the time for your guide update.
Do not choose the top of the hour, the bottom of the hour, and the quarter hours especially during the overnight hours around midnight +/- 4 hours. Plus on the East Coast and minus on the West. This seems to be the busiest times for SD and they are probably being overload.

[NaivekGB]

Cool thanks! Changed it to a different time...

[IT Troll]

Version: 1.7.1.0 Changelog

• added more filtering options in client primarily for DVB users (TV, Radio, Interactive TV, Encrypted, Unencrypted, Suggested Blocked and not Suggested Blocked)
• improved enabling/disabling channels particularly for DVB-T

Thanks, these features came in handy after the recent channel renumbering of Freeview in the UK. I had a good clear out of all the old channels and also many of those which were suggested blocked. The latter being mostly partial reception from alternative transmitters.

[garyan2]

You can thank yourself too, a bit. I used a backup you sent me a ways back to make sure it worked well with DVB-T and these "recommended blocked" tuner channels.

Some information for everyone. For DVB-T scans, WMC will scan the UHF channels and if there is anything to tune in, it will include the channel frequency as well as +/-167kHz. So if you have 4 tuners, each merged channel will have 12 sources in WMC (4 for each frequency). Those frequencies that work are what WMC will use for watching and recording... the others will be flagged as recommended blocked and initially be disabled. The thing to know though is, if you enable/disable the channel within WMC, it overrides that recommendation so that all 12 sources will be enabled/disabled together. Now you have a merged channel that may try to tune to the wrong frequency initially until it finds a signal on one of the available sources. If you enable/disable the channels in the epg123 client, it respects the recommended block flag and will not override anything so that only the sources that "work" are the ones that are enabled or disabled.

[IT Troll]

That is interesting info about the +/-167kHz. I have obviously witnessed that but didn’t realise it was intended behaviour. I have spent much time in the past reordering the priority on sources so the correct ones are first in the list.