Unable to complete task error popup every 2 minutes

Technical Support
1

Note - If you do not fill in every section below, your post won’t be answered - you must provide the steps you have followed so far and the actions you’ve already taken. Make sure to remove this notice from your post too.

Screenshots of text are prohibited. Always copy and paste text as text.

Make sure you search before posting! Duplicate posts for the same issue may not be answered.

OS Name/Version: Windows Server 2022 (Desktop)

Product Name/Version: AMP 2.6.0.6 in Controller/Target mode

Problem Description:

Experiencing a strange issue where every 2 minutes a popup will appear stating:

Unable to complete task
This task could not be completed: (Remote on Blackpearl) Updating remote sources - Fetching latest configurations… - State:3.
Response status code does not indicate success: 502 (Bad Gateway).

This will start to pop up every 2 minutes when it keeps trying to fetch the configurations for that instance. “Blackpearl” is the name of the target AMP node, also running Windows Server 2022. Controller and target both are running Windows Server 2022. Sometimes the error has said error 400 as well as 502.
What’s strange is a reboot of the ADS instance on the target resolves it, and fetching configurations works OK for an unknown amount of time (I think a few hours) and then it will start displaying this error again.

I enabled debug logging and see this in the deployment log for the target node every 2 minutes with the popup error:

:AMP_SYSTEM

17:31:00
Changing setting ADSModule.Defaults.DefaultAuthServerURL to http ://ampcntrl01. MYDOMAIN. net:8080/
Changing setting ADSModule.ADS.ConfigurationRepositories to [“CubeCoders/AMPTemplates”]

17:31:01
Updating remote source CubeCoders/AMPTemplates
Downloading https ://github.com/CubeCoders/AMPTemplates/archive/refs/heads/main.zip, saving to CubeCoders-AMPTemplates.zip…
Failed Download: https ://github.com/CubeCoders/AMPTemplates/archive/refs/heads/main.zip
HttpRequestException
[0] (HttpRequestException) : Response status code does not indicate success: 502 (Bad Gateway).
at Net.Http.HttpResponseMessage.EnsureSuccessStatusCode() at ModuleShared.Utilities.DownloadFileWithProgressAsync(String url, String filename, RunningTask task, Boolean autoRemoveTask, String UserAgent)
Failed to update CubeCoders/AMPTemplates

Steps to reproduce:

Start up the controller and target instance
Wait a couple of hours.
The pop up will start to appear every 2 minutes. A reboot of the ADS01 instance on the target node will resolve it for another couple of hours.

Actions taken to resolve so far:
Reviewed logs to get any other specific errors. Did not find anything other than above.
Ran the steps here: discourse .cubecoders. com/t/something-not-working-on-windows-try-the-following/241
Repaired the datastores using the button on controller and when managing target node.

2

Looks to be an error on GitHub’s end, considering that 502 is an internal server error for the webserver you’re reaching.

A sanity check on your end may be to check that DNS is working as expected on your machine and that you can ping github.com on that system and/or view it using a web browser

3

Could it be rate limiting or something similar? Any way to change the frequency of the remote sources update from 2 minutes? I could do with daily even.
It’s strange that it works fine with no errors for a while after a reboot then starts back up after a couple of hours.
Github.com resolves correctly it seems.

4

Rate limiting would be a 403 Forbidden if my memory serves right (4xx errors are client side issues, 5xx are server-side).

Are you restarting the whole machine, or just running ampinstmgr restart ADS01 ?

5

Just restarting the ADS01 instance service from the windows services UI.

6

Found this that seems sorta relevant, googling around just returns oodles of generic “no workee” queries about people using a browser, not with code lol
https://stackoverflow.com/questions/44594353/the-requested-url-returned-error-502-corporate

Checked my settings and couldn’t find any way to change the interval. Does it limit you in the panel in any way when it’s erroring out? Thinking you could get away with restarting the ADS just when you go to create new/update instances.

Though just curious, if you could find any of the 4xx responses in the logs I’d like to take a peek at those. Could maybe be a breadcrumb as to why it throws a 502

7

Nothing really seems to be impacted. Everything still seems to work fine even creating instances (it just can’t pull the latest config when it does).
The popup is just pretty annoying every 2 minutes. Although a scheduled reboot of the ADS01 instance service might work as a workaround since it seems to clear it up each time.

Here is a success log entry right after the instance rebooted (I think in this case the whole system rebooted for Windows updates) the other day on 12/31/2024:

[03:40:43] [Core Debug/5] : Login request from 10.151.12.109 for AMP_SYSTEM
[03:40:43] [Core Activity/5] : Authentication attempt for user AMP_SYSTEM from 10.151.12.109
[03:40:43] [Core Debug/5] : Logging in via remote auth server http ://ampcntrl01.MYDOMAIN.net:8080/…
[03:40:43] [Core Activity/11] : Authentication success
[03:40:43] [Core Debug/11] : Creating new session for AMP_SYSTEM at 10.151.12.109 (CCL/AMPAPI) API Login
[03:40:43] [Core Debug/11] : Total sessions count: 0
[03:40:43] [Core Debug/11] : Timeout: 15
[03:40:43] [Core Debug/11] : Login success from 10.151.12.109 assigned session ID cdb1e146-00f0-4bce-b8b8-2427ef0f3cbb
[03:40:43] [Core:AMP_SYSTEM Activity/5] : Changing setting ADSModule.Defaults.DefaultAuthServerURL to http ://ampcntrl01.MYDOMAIN.net:8080/
[03:40:43] [Core:AMP_SYSTEM Activity/11] : Changing setting ADSModule.ADS.ConfigurationRepositories to [“CubeCoders/AMPTemplates”]
[03:40:43] [ADS:AMP_SYSTEM Info/5] : Updating remote source CubeCoders/AMPTemplates
[03:40:43] [Core:AMP_SYSTEM Debug/5] : Downloading https ://github.com/CubeCoders/AMPTemplates/archive/refs/heads/main.zip, saving to CubeCoders-AMPTemplates.zip…
[03:40:43] [Core:AMP_SYSTEM Debug/10] : Moving C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Temp\akkxemm4.yvc to CubeCoders-AMPTemplates.zip…
[03:40:48] [Core:AMP_SYSTEM Debug/5] : Task Updating remote sources (Fetching latest configurations…) ended: Finished

This continues working fine for about 18 hours it seems then the first time it encounters an error it’s an error 400:

[21:36:00] [System:Anonymous Debug/58] : Login request from 10.151.12.109 for AMP_SYSTEM
[21:36:00] [System:Anonymous Activity/58] : Authentication attempt for user AMP_SYSTEM from 10.151.12.109
[21:36:00] [System:Anonymous Debug/58] : Logging in via remote auth server http ://ampcntrl01.MYDOMAIN.net:8080/…
[21:36:00] [System:Anonymous Activity/58] : Authentication success
[21:36:00] [Core:Anonymous Debug/58] : Ending session d154bf38-2b5a-4e77-92c6-6013291f1672 (AMP_SYSTEM at 10.151.12.109, API Login) - No activity for 15 minutes.
[21:36:00] [Core:Anonymous Debug/58] : Creating new session for AMP_SYSTEM at 10.151.12.109 (CCL/AMPAPI) API Login
[21:36:00] [Core:Anonymous Debug/58] : Total sessions count: 7
[21:36:00] [Core:Anonymous Debug/58] : Timeout: 15
[21:36:00] [Core:Anonymous Debug/58] : Login success from 10.151.12.109 assigned session ID 34c9438b-506f-423d-8bd4-c29633ea798d
[21:36:00] [Core:AMP_SYSTEM Activity/9] : Changing setting ADSModule.Defaults.DefaultAuthServerURL to http ://ampcntrl01.MYDOMAIN.net:8080/
[21:36:01] [Core:AMP_SYSTEM Activity/58] : Changing setting ADSModule.ADS.ConfigurationRepositories to [“CubeCoders/AMPTemplates”]
[21:36:01] [System:AMP_SYSTEM Info/9] : Updating remote source CubeCoders/AMPTemplates
[21:36:01] [Core:AMP_SYSTEM Debug/9] : Downloading https ://github.com/CubeCoders/AMPTemplates/archive/refs/heads/main.zip, saving to CubeCoders-AMPTemplates.zip…
[21:36:01] [Core:AMP_SYSTEM Debug/9] : Failed Download: https ://github.com/CubeCoders/AMPTemplates/archive/refs/heads/main.zip
[21:36:01] [Core:AMP_SYSTEM Error/9] : HttpRequestException
[21:36:01] [Core:AMP_SYSTEM Error/9] : [0] (HttpRequestException) : Response status code does not indicate success: 400 (Bad Request).
[21:36:01] [Core:AMP_SYSTEM Error/9] : at Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
at ModuleShared.Utilities.DownloadFileWithProgressAsync(String url, String filename, RunningTask task, Boolean autoRemoveTask, String UserAgent)
[21:36:01] [ADS:AMP_SYSTEM Warning/9] : Failed to update CubeCoders/AMPTemplates
[21:36:06] [Core:AMP_SYSTEM Debug/9] : Task Updating remote sources (Fetching latest configurations…) ended: Finished
[21:36:06] [Core:AMP_SYSTEM Debug/58] : Task Updating remote sources (Fetching latest configurations…) ended: Finished

It then continues to give the same error 400 every 2 minutes for the next few days until 01/03/2025 when the first error 502 comes in and stays after that:

[01:58:00] [System:Anonymous Debug/47] : Login request from 10.151.12.109 for AMP_SYSTEM
[01:58:00] [System:Anonymous Activity/47] : Authentication attempt for user AMP_SYSTEM from 10.151.12.109
[01:58:00] [System:Anonymous Debug/47] : Logging in via remote auth server http ://ampcntrl01.MYDOMAIN.net:8080/…
[01:58:00] [System:Anonymous Activity/22] : Authentication success
[01:58:00] [Core:Anonymous Debug/22] : Creating new session for AMP_SYSTEM at 10.151.12.109 (CCL/AMPAPI) API Login
[01:58:00] [Core:Anonymous Debug/22] : Total sessions count: 8
[01:58:00] [Core:Anonymous Debug/22] : Timeout: 15
[01:58:00] [Core:Anonymous Debug/22] : Login success from 10.151.12.109 assigned session ID bfd164ee-4d60-4567-b4e6-4812bcde69e0
[01:58:00] [Core:AMP_SYSTEM Activity/47] : Changing setting ADSModule.Defaults.DefaultAuthServerURL to http ://ampcntrl01.MYDOMAIN.net:8080/
[01:58:00] [Core:AMP_SYSTEM Activity/22] : Changing setting ADSModule.ADS.ConfigurationRepositories to [“CubeCoders/AMPTemplates”]
[01:58:00] [System:AMP_SYSTEM Info/3] : Updating remote source CubeCoders/AMPTemplates
[01:58:00] [Core:AMP_SYSTEM Debug/3] : Downloading https ://github.com/CubeCoders/AMPTemplates/archive/refs/heads/main.zip, saving to CubeCoders-AMPTemplates.zip…
[01:58:00] [Core:AMP_SYSTEM Debug/3] : Failed Download: https ://github.com/CubeCoders/AMPTemplates/archive/refs/heads/main.zip
[01:58:00] [Core:AMP_SYSTEM Error/3] : HttpRequestException
[01:58:00] [Core:AMP_SYSTEM Error/3] : [0] (HttpRequestException) : Response status code does not indicate success: 400 (Bad Request).
[01:58:00] [Core:AMP_SYSTEM Error/3] : at Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
at ModuleShared.Utilities.DownloadFileWithProgressAsync(String url, String filename, RunningTask task, Boolean autoRemoveTask, String UserAgent)
[01:58:00] [ADS:AMP_SYSTEM Warning/3] : Failed to update CubeCoders/AMPTemplates
[01:58:05] [Core:AMP_SYSTEM Debug/3] : Task Updating remote sources (Fetching latest configurations…) ended: Finished
[01:58:05] [Core:AMP_SYSTEM Debug/22] : Task Updating remote sources (Fetching latest configurations…) ended: Finished
[02:01:00] [System:Anonymous Debug/22] : Login request from 10.151.12.109 for AMP_SYSTEM
[02:01:00] [System:Anonymous Activity/22] : Authentication attempt for user AMP_SYSTEM from 10.151.12.109
[02:01:00] [System:Anonymous Debug/22] : Logging in via remote auth server http ://ampcntrl01.MYDOMAIN.net:8080/…
[02:01:00] [System:Anonymous Activity/22] : Authentication success
[02:01:00] [Core:Anonymous Debug/22] : Ending session 55a0afc8-4ff7-46b2-8e64-26685f4a0312 (AMP_SYSTEM at 10.151.12.109, API Login) - No activity for 15 minutes.
[02:01:00] [Core:Anonymous Debug/22] : Ending session d101535c-312f-4458-a762-302cdc2ebf3c (AMP_SYSTEM at 10.151.12.109, API Login) - No activity for 15 minutes.
[02:01:00] [Core:Anonymous Debug/22] : Creating new session for AMP_SYSTEM at 10.151.12.109 (CCL/AMPAPI) API Login
[02:01:00] [Core:Anonymous Debug/22] : Total sessions count: 7
[02:01:00] [Core:Anonymous Debug/22] : Timeout: 15
[02:01:00] [Core:Anonymous Debug/22] : Login success from 10.151.12.109 assigned session ID 44f648b4-085e-4721-b30c-e61e335874ea
[02:01:00] [Core:AMP_SYSTEM Activity/22] : Changing setting ADSModule.Defaults.DefaultAuthServerURL to http ://ampcntrl01.MYDOMAIN.net:8080/
[02:01:00] [Core:AMP_SYSTEM Activity/33] : Changing setting ADSModule.ADS.ConfigurationRepositories to [“CubeCoders/AMPTemplates”]
[02:01:00] [System:AMP_SYSTEM Info/33] : Updating remote source CubeCoders/AMPTemplates
[02:01:00] [Core:AMP_SYSTEM Debug/33] : Downloading https ://github.com/CubeCoders/AMPTemplates/archive/refs/heads/main.zip, saving to CubeCoders-AMPTemplates.zip…
[02:01:00] [Core:AMP_SYSTEM Debug/33] : Failed Download: https ://github.com/CubeCoders/AMPTemplates/archive/refs/heads/main.zip
[02:01:00] [Core:AMP_SYSTEM Error/33] : HttpRequestException
[02:01:00] [Core:AMP_SYSTEM Error/33] : [0] (HttpRequestException) : Response status code does not indicate success: 502 (Bad Gateway).
[02:01:00] [Core:AMP_SYSTEM Error/33] : at Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
at ModuleShared.Utilities.DownloadFileWithProgressAsync(String url, String filename, RunningTask task, Boolean autoRemoveTask, String UserAgent)
[02:01:00] [ADS:AMP_SYSTEM Warning/33] : Failed to update CubeCoders/AMPTemplates
[02:01:05] [Core:AMP_SYSTEM Debug/33] : Task Updating remote sources (Fetching latest configurations…) ended: Finished

8

Yeah that’s a weird one for sure, though the 400s do lean towards your rate limit theory.
Here’s what I think might be going on:

  • AMP is fine for awhile
  • Something inside AMP changes slightly while running, so the query sent to GH is malformed
  • GH responds with “Bad Request”
  • Repeat until GH is fed up and blocks the request at a higher level

You already have debug logging on, so not too sure what else to check.
Another user awhile back said they fiddled with their firewall and that resolved it

9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.