Question concerning v1.7.0.x

[garyan2]

Question for those users that have downloaded and installed v1.7.0.x. Any perceived issues? This version installs a service that will run on bootup that acts as a proxy when requesting images, Automatically retrieves a new token if the response from SD is an error for an UNKNOWN_USER, and has the ability to cache those images locally to serve out to any number of client machines on your network. It also allows you to view the trace.log file, a new server.log file, and download the MXF and XMLTV files.

http://<HOSTNAME>:9009/trace.log
http://<HOSTNAME>:9009/server.log
http://<HOSTNAME>:9009/output/epg123.mxf
http://<HOSTNAME>:9009/output/epg123.xmltv

If you've given it a go, please provide some information on how it is working and how you are using it... either as Full Mode install (everything local to that machine), or acting as a server to clients with v1.7.0.x as well or some XMLTV consuming software.

I'm just looking for input whether I need to tweak some things, or it is good for a full release. I've been running it on a 24/7 machine without any issues and it is properly refreshing the token when it needs to. I have also been using the cache which does pretty nicely for the clients.

[d00zah]

Full Mode install.

v1.7.0.0 was a bust (no images at all).
v1.7.0.1 has been rock solid since release. No missing images, or other detectable issues.

IF portable install is a thing of the past, may I request(?) the following be made optional:

- Create desktop icon (already an option)
- Create start menu group option (I relocate it)
- Create traytool startup option (I don't use it)

Have been using the portable version to avoid having to go back & undo these, post update... OCD

Thanks, Gary! Great work!!

[cmal]

Hi
I have not yet tested 1.7. but have a general question: I have one main machine and one client only; the main machine shall not run all the time but is (over night) also in stand-by...
if the image cache is on that main machine then this means the client can only access the pictures if the main PC is running and the client has access to the cache, correct?
if that is the case, then the question would be whether the image-cache could also be placed e.g. on a NAS... otherwise the client install is not meaning full anymore (from my point of view).
Best Regards

[alhaunts]

Running 1.7.0.1 for about a week now. Windows 7 64 bit in Full Mode. Like d00zah says, solid all around. Only thing I've noticed is this log entry most days:

[9/2/2021 4:57:00 AM] [ INFO] Beginning EPG123 update execution. 2021-09-02 08:57:00Z
[9/2/2021 4:57:00 AM] [ INFO] DaysToDownload: 14 , TheTVDBNumbers : True , PrefixEpisodeTitle: False , PrefixEpisodeDescription : False , AppendEpisodeDesc: False , OADOverride : True , TMDbCoverArt: True , IncludeSDLogos : True , AutoAddNew: True , CreateXmltv: False , ModernMediaUiPlusSupport: False
[9/2/2021 4:57:00 AM] [ INFO] SD API WebException Thrown. Message: The remote server returned an error: (403) Forbidden. , Status: ProtocolError
[9/2/2021 4:57:00 AM] [ INFO] SD responded with error code: 4003 , message: Invalid username or token has expired. , serverID: 20141201.web.1 , datetime: 2021-09-02T08:57:00Z
[9/2/2021 4:57:00 AM] [ INFO] Validation of cached token failed. Requesting new token.
[9/2/2021 4:57:01 AM] [ INFO] Token request successful. serverID: 20141201.web.1 , datetime: 2021-09-02T08:57:01Z
[9/2/2021 4:57:01 AM] [ INFO] Status request successful. account expires: 2021-12-01T16:41:19Z , lineups: 1/4 , lastDataUpdate: 2021-09-02T00:09:07Z
[9/2/2021 4:57:01 AM] [ INFO] system status: Online , message: No known issues.

I'm guessing the issue is on SD's end. Doesn't seem to affect functionality at all.

[d00zah]

Running 1.7.0.1 for about a week now. Windows 7 64 bit in Full Mode. Like d00zah says, solid all around. Only thing I've noticed is this log entry most days:

[9/2/2021 4:57:00 AM] [ INFO] SD API WebException Thrown. Message: The remote server returned an error: (403) Forbidden. , Status: ProtocolError
[9/2/2021 4:57:00 AM] [ INFO] SD responded with error code: 4003 , message: Invalid username or token has expired. , serverID: 20141201.web.1 , datetime: 2021-09-02T08:57:00Z
[9/2/2021 4:57:00 AM] [ INFO] Validation of cached token failed. Requesting new token.

This looks like one of the issues this release addresses. Did you select the recommended options on the configuration GUI > 'Service' tab?

[Space]

I don't think it is a problem with the recommended options. This looks more like EPG123 assuming you already have a valid token and only requesting a new token if it gets a "token expired" response when trying to perform an operation (and then retrying the original operation).

This can be avoided (most of the time) by keeping track of how long ago you last requested a token and if the existing token should still be valid, trying the operation first and only requesting the token if it fails (due to token expiration). If the token should be expired (because an hour has elapsed since you got the last token) then you can request a new token first, before you perform the operation.

If you don't know how long the existing token is going to be valid, or you don't want to bother keeping track of when the last token is going to expire, then you can just do what EPG123 appears to be doing now: try the operation, if it fails due to expired token then get a new token and try the operation again.

Unless the SD API includes a way of knowing when the new token will expire (rather than just "knowing" it will expire in a hour), the current way is probably the best way. Since the token validity period may change in the future, hard coding a 1 hour period doesn't seem like the best option.
Also, you'd have to coordinate between the EPG123 client and the EPG123 daemon/HTTP server as to what the current token is and how much longer it would be valid, which would be another complication.

[alhaunts]

On the Service tab I selected "Automatically refresh token" but not "Enable image caching" since I don't have any clients hanging off my two machines. Both are Full installs as I don't want one to be dependent on the other being awake or powered-up. The weird thing is this "token has expired" thing only happens sometimes during the daily scheduled update, not every time.

[garyan2]

Running 1.7.0.1 for about a week now. Windows 7 64 bit in Full Mode. Like d00zah says, solid all around. Only thing I've noticed is this log entry most days:

[9/2/2021 4:57:00 AM] [ INFO] Beginning EPG123 update execution. 2021-09-02 08:57:00Z
[9/2/2021 4:57:00 AM] [ INFO] DaysToDownload: 14 , TheTVDBNumbers : True , PrefixEpisodeTitle: False , PrefixEpisodeDescription : False , AppendEpisodeDesc: False , OADOverride : True , TMDbCoverArt: True , IncludeSDLogos : True , AutoAddNew: True , CreateXmltv: False , ModernMediaUiPlusSupport: False
[9/2/2021 4:57:00 AM] [ INFO] SD API WebException Thrown. Message: The remote server returned an error: (403) Forbidden. , Status: ProtocolError
[9/2/2021 4:57:00 AM] [ INFO] SD responded with error code: 4003 , message: Invalid username or token has expired. , serverID: 20141201.web.1 , datetime: 2021-09-02T08:57:00Z
[9/2/2021 4:57:00 AM] [ INFO] Validation of cached token failed. Requesting new token.
[9/2/2021 4:57:01 AM] [ INFO] Token request successful. serverID: 20141201.web.1 , datetime: 2021-09-02T08:57:01Z
[9/2/2021 4:57:01 AM] [ INFO] Status request successful. account expires: 2021-12-01T16:41:19Z , lineups: 1/4 , lastDataUpdate: 2021-09-02T00:09:07Z
[9/2/2021 4:57:01 AM] [ INFO] system status: Online , message: No known issues.

I'm guessing the issue is on SD's end. Doesn't seem to affect functionality at all.

When epg123 retrieves a token from SD, it will store the expiration time in the registry along with the token. When it runs an update, it will look at that time and if it is expected to expire within 1 hour it will request a new one; otherwise it will try to validate the token by requesting a user status. If that fails with a 4003 message, then it will go ahead and request a new token. Also, since there can be only 1 valid token at a time, if another program or another machine grabbed a new token, it will invalidate the token this machine holds and you will get the same result.

Really, this is just epg123 checking to see if it would be possible to use the last token it stored in the registry and if not then get a new one. That is why the entries are INFO and not ERROR. No harm no foul.

[garyan2]

Hi
I have not yet tested 1.7. but have a general question: I have one main machine and one client only; the main machine shall not run all the time but is (over night) also in stand-by...
if the image cache is on that main machine then this means the client can only access the pictures if the main PC is running and the client has access to the cache, correct?
if that is the case, then the question would be whether the image-cache could also be placed e.g. on a NAS... otherwise the client install is not meaning full anymore (from my point of view).
Best Regards

Unfortunately, that is correct for right now. If your server component is not on 24/7, then you will not be able to retrieve new images. The MXF file created has all the images linked to that server, and the server holds the "current" token from SD. If it is off, you will just get timeouts.

EDIT: I should clarify that the machine doesn't need to be on 24/7, but certainly needs to be on when you need to request images from the client.

[cmal]

...if another program or another machine grabbed a new token, it will invalidate the token this machine holds...

is there a limit to how many times this token can be renewed per time-frame? If, as in my case, two independently used PCs are scanning the EPG at the same time, then the token has to be renewed with high frequency. Does SD tolerate this? If not, then this change of policy means effectively : this is a single install/machine license...

[garyan2]

This is where is gets kind of confusing. There can only be 1 active token for downloading data, but for downloading images it just needs a successful login/token request within the last 24 hours. It is this part that isn't working correctly on the SD side of things in that your "expiration" may be based on an old token, not the last token, and is one of the reasons I included the automatic token refresh.

So there really is not a lot of tokens being requested in the day even if you have multiple machines using the server side of epg123.

New token: When requesting data for your scheduled update and the remembered token is either expired, about to expire, or is no longer valid.
New token: When requesting an image and the response is UNKNOWN_USER, meaning access to images has expired. The new token here will reset the clock for image access to 24 hours.

Reuse token: When requesting data for your scheduled update and the remember token does not expire for at least 1 hour and is still valid.
Reuse token: Image requests up until the UNKNOWN_USER response.

In the future, when image requests will require an actual token, it is planned that each user will be able to "hold" up to 5 valid tokens. Requesting a new token will invalidate the oldest.

[Gripweed]

Admittedly I'm having a problem understanding this whole concept. EPG123 used to be as easy as, well, 1-2-3. I'm seriously thinking of dropping WMC/SD/EPG123 for Emby or some other PVR to simplify the process. And as mentioned before, I rarely ever use WMC as a media server or actually look at the Guide so my needs for image updates are few and far between. My concern is that there might be a security risk in this proxy service you've created. Does it mean that my port 9009 is open to the world? Or do I need to port forward the port in my router to get this to work?

[garyan2]

It is still that easy. There is nothing you need to do for this service to work, so install and walk away... nothing spectacular or even new as far as the install and operation. The technical side of the change is basically the image links in the MXF file will be pointing to the service rather than Schedules Direct. This way your WMC cache will work properly and you won't be downloading any images repeatedly because SD redirects the image requests. Since you "rarely" use WMC or view the guide, you won't even be doing that very often.

Port 9009 will only be open to the outside world if you open it up in your router and forward it to the computer with the service running. You don't need to do anything with your router for it to work on your local network.

How it works is instead of having a link to https ://schedulesdirect.org/image/0123456789abcdef.jpg in the MXF file for a program/series, it will be a link to http://<HOSTNAME>:9009/image/0123456789abcdef.jpg. The service will then request the image from the Schedules Direct server (original link) and provide that to the client. If the image is cached, it will just send the file to the client without requesting it from Schedules Direct. That's it. If you never request an image, or even request the MXF file or XMLTV file, then the service is just sitting there doing nothing. You of course can disable the service and the only thing that will be different than right now is that you will not get any images from SD... note you will still get images from TMDB if you have that option enabled which will be a very small number of images.

[StinkyImp]

It is still that easy. There is nothing you need to do for this service to work, so install and walk away...

You of course can disable the service and the only thing that will be different than right now is that you will not get any images from SD...

I'm "almost" understanding this, which is most likely the result of Alzheimer's Old-Timer's disease.

Soooo.... In order to receive images, the server must be running, otherwise no images at all?

Even for those of us running standalone installations and once-a-day scheduled SD updates?

I think that last question is where I'm not getting it. My understanding is that during my regularly scheduled update I receive a token and then use that during my update to receive images directly from SD using that session's "valid" token. 24 hours later I get a new token for the now current update. Rinse, wring, repeat. If so, why would I need to cache them in the local server or even run the server at all?

[IT Troll]

My understanding is that during my regularly scheduled update I receive a token and then use that during my update to receive images directly from SD using that session's "valid" token. 24 hours later I get a new token for the now current update. Rinse, wring, repeat. If so, why would I need to cache them in the local server or even run the server at all?

I might be wrong, but I think normally only the image URL is received during the scheduled update. The image is not actually downloaded until first accessed in the Media Center EPG, which may be some time later and when the token has expired. The proxy server acts as a middle man to ensure you have a valid token at the point you download an image.

I've not actually installed the new version yet. These are big changes and I don't have the time at the moment to troubleshoot if there happen to be any problems. Has anyone else in the UK tried the new version? I'd be more comfortable giving it a go if other UK members have had success.

[garyan2]

You don't need to cache the images at all, just it is beneficial if you have multiple clients. The big role of the service is to ensure you can download images from SD, and to ensure any images you download from SD only need to be downloaded once.

Current method:
1. WMC requests image from SD directly. (https ://json.schedulesdirect.org/20141201/image/0123456789abcdef.jpg)
2. SD responds with a redirect to amazon server with authorization tokens that expire in 2 minutes (https ://scheduledirect-api20141201a.s3.amazonaws.com/0/0123456789abcdef.jpg?X-Amx-Content-Sha256.......)
3. WMC caches the image using the redirected link in step 2 in your internet cache.
[Close the WMC session and open again...]
4. WMC requests image from SD directly for same image because that image/URL combination does not exist in internet cache, only the image/redirected URL is in cache.
5. SD responds with a redirect to amazon server, but it is different than step 2 since the authorization in the URL changed.
6. WMC downloads and caches the image again in internet cache which is only good until you close WMC.

So basically, WMC will download all images requested, every WMC session, since none of the cached images are "valid" anymore. This is the reason currently that if your access to images has expired and you open WMC, you will have no images at all even though you could have downloaded all of the images in the last session.

Need for service:
1. WMC requests image from service. (http ://<HOSTNAME>:9009/image/0123456789abcdef.jpg)
2. Service requests image from SD while adding a token query value. (https ://json.schedulesdirect.org/20141201/image/0123456789abcdef.jpg?token=<valid token>)
3. Service downloads image from SD and provides to WMC which will cache the image using the URL from step 1.
[Close the WMC session and open again...]
4. WMC has the image/URL from step 1 in cache so will not need to download it again.

So the service does 2 things. It keeps consistency for your WMC internet cache to prevent downloading the same image over and over again. It will also add a valid token in the query for SD to authorize download of the image. The token currently is not required, but will be in the future.

[garyan2]

My understanding is that during my regularly scheduled update I receive a token and then use that during my update to receive images directly from SD using that session's "valid" token. 24 hours later I get a new token for the now current update. Rinse, wring, repeat. If so, why would I need to cache them in the local server or even run the server at all?

I might be wrong, but I think normally only the image URL is received during the scheduled update. The image is not actually downloaded until first accessed in the Media Center EPG, which may be some time later and when the token has expired. The proxy server acts as a middle man to ensure you have a valid token at the point you download an image.

I've not actually installed the new version yet. These are big changes and I don't have the time at the moment to troubleshoot if there happen to be any problems. Has anyone else in the UK tried the new version? I'd be more comfortable giving it a go if other UK members have had success.

You are correct on the first part.

For the second, location doesn't matter. All the service is doing is handling the request/download of images from SD so that the redirects and tokens are transparent to WMC, thus ensuring you can take full advantage of your internet cache and significantly reduce the number of images requested to SD.

[Space]

It seems that the "current method" is inefficient even if it did not have the issue with the 2 minute token expiring.

Even if the image exists in the cache, WMC still has to use resources to query the URL from SD. Only after it gets the redirect from that query does it see that it has a cached version of that redirect and uses that instead of downloading the image.

This is of course made worse by the 2-minute token, because the redirect URL it would get would be different than the cached URL, so it has to not only download the image again, it also now has two copies of that image cached (although WMC may be smart enough to get rid of the old one due to the original URL being the same). This apparently can happen every time it wants to display an image, so you can end up with multiple copies of the same image in the cache.

I do have one question... with the service method, does it ever check to see if a WMC cached image has changed and download the new version if it has? Or does it periodically invalidate the cache so that it is forced to re-download the image again (potentially an updated image)? I am assuming it does because otherwise you would end up with stale images.

Does the EPG123 service cache also have a mechanism of updating stale images?

[garyan2]

I do have one question... with the service method, does it ever check to see if a WMC cached image has changed and download the new version if it has? Or does it periodically invalidate the cache so that it is forced to re-download the image again (potentially an updated image)? I am assuming it does because otherwise you would end up with stale images.

Does the EPG123 service cache also have a mechanism of updating stale images?

Absolutely correct on the inefficiency of the current model. The service will take care of that.

For versions 1.7.0.0/1, the images downloaded and cached by WMC never expire (this is the way it has always been even pre-epg123). The images in the epg123 cache will expire 30 days after the last request from a client. I'm working on changing that now. The images provided by SD have a 30-day lifespan from when it was created, so there is an expiry date for each image. I'm going to cascade that into the epg123 cache as well as the image WMC downloads in the internet cache. As long as WMC respects the Expires field, when the image expires on the SD side, it will expire in WMC and the epg123 cache forcing a refresh when it is requested again.

[IT Troll]

For the second, location doesn't matter.

It is not so much the download location that is a concern. Rather that some regional formatting or language setting might unexpectedly trip things up in the new code.