Converting CMG to virtual machine scale set failed

[Bram]

The conversion of a classic CMG to the virtual machine scale set model failed. In the end I found out why: Microsoft.KeyVault was not yet registered as a resource provider in the Azure subscription...
But now we're stuck with a broken CMG that has status 'error' - unable to connect to the cloud service.

The broken CMG service name uses our own domain and wildcard certificate: brokencmg.company.com. Just wondering if I can just spin up a new working cmg under the same domain newcmg.company.com and then just point the CNAME record of the broken one to the new cloudapp service in Azure so both URLs keep working?

[anyweb]

what does the connection analyzer tell you ?

[Bram]

Everything is green except for the first item 'Check the CMG Service is in a ready state'. State of the CMG Service is '2'.

CloudMgr.log is showing following issues: 
ERROR: Service brokencmg does not exist.
ERROR: Exception occured trying to change service status brokencmg: System.InvalidOperationException: VM scale set does not exist for service.~~   at Microsoft.ConfigurationManager.CloudServicesManager.ChangeDeploymentStatusTask.StartDeployment().
ERROR: TaskManager: Task [ChangeDeploymentStatus for service brokencmg] has failed. Exception System.InvalidOperationException, VM scale set does not exist for service..

In the console, the deployment model of the broken CMG shows 'Virtual machine scale set' while the conversion did not complete and the old classic services are still there in Azure.

 

[anyweb]

can you share the cloudmgr.log with me, feel free to remove any references to your company in the log.

[anyweb]

ok an update on this, if you want it working right now then I'm afraid you'll have to delete it and start again from scratch (including uploading all the content)

don't mess with cnames it's not supported, you might get it to work, but it's not supported so don't bother.

I've given your experience as feedback to the Microsoft product group and they are taking the feedback seriously, sorry for the hassles...

 

[FlorianK]

I actually have a similar issue with a failed CMG

CMG not reachable - Microsoft Q&A

A lot of clients are only connected through CMG and doesn't use VPN. Any idea how I can move my disconnected clients to the new CMG?

[anyweb]

well those clients would have been using the working CMG before it broke during conversion right ? and that's where they are getting their policy

so if the CMG is down (broken) they cannot get new policy, so you'll have to get creative in terms of how to target those clients,

see below and linked here for some suggestions, but you'll need a working CMG before trying these so you'll need to stand up a new working CMG...

"Once the cloud management gateway (CMG) and the supporting site system roles are operational, you may need to make configuration changes on Configuration Manager clients.

Clients that can communicate with the management point automatically get the location of the CMG service on the next location request. The polling cycle for location requests is every 24 hours. If you don't want to wait for the normally scheduled location request, you can force the request. To force the request, restart the SMS Agent Host service (ccmexec.exe) on the computer.

For devices that aren't connected to the internal network, there are several options to configure them with a CMG location. For more information, see Install off-premises clients using a CMG.

Note

By default all clients receive CMG policy. Control this behavior with the client setting, Enable clients to use a cloud management gateway. For more information, see About client settings."

[Bram]

On 2/25/2022 at 8:18 PM, anyweb said:

ok an update on this, if you want it working right now then I'm afraid you'll have to delete it and start again from scratch (including uploading all the content)

don't mess with cnames it's not supported, you might get it to work, but it's not supported so don't bother.

I've given your experience as feedback to the Microsoft product group and they are taking the feedback seriously, sorry for the hassles...

 

Thanks, for submitting feedback at MS, really appreciated.
I've deployed a new CMG now which is indeed probably the easiest solution.

[KeyboardSmasher]

Any feedback from MS on this? We are in the exact same position after attempting a conversion from classic to VMSS.

"System.InvalidOperationException: VM scale set does not exist for service."

If its still the case we need to delete and recreate - are there any side effects of simply re-deploying another classic CMG with the same service name?

This could get us out of trouble and we could then deploy a scale set in parallel and migrate to it.

 

[anyweb]

I would not do anything unsupported, I've pinged Microsoft again for comment, if they come back to me i'll let you know

[KeyboardSmasher]

For anyone in the same situation - We simply deleted the failed classic CMG and recreated a new Scale Set CMG using the same service name and certificate.  

Changed DNS to point to the new URL and all worked fine. Clients reconnected to the new CMG without any issues.

[anyweb]

thanks for the update, hopefully it helps someone in the same position

[dan.clark@stryker.com]

Had a similar issue with a conversion from Cloud service (classic) to Virtual machine scale set failing due to a custom tagging policy being enforced in the Azure tenant.

 

Found this blog that offered a workaround for reverting back to the original CMG config. https://rui-qiu.com/sccm/fix-cmg-conversion-failed-from-classic-to-vm-scale-set/

 

"So I contacted Microsoft, the fix is the manually revert back to CMG classic state inside SCCM console, and do the conversion again. Here are the steps:

1. Stop SMS Executive service

2. Run SQL query on your SQL Server: update azure_service set FQDN = 'xxx.cloudapp.net', DeploymentModel = '1'

3. Start SMS Executive service

4. Monitor cloudmgr.log, wait for CMG convert back to Classic"

 

I was able to use that SQL query to get my CMG back online in the Cloud service (classic) mode while I work through the security policy that is blocking my upgrade.