Repository navigation
Add settings retry attempts and interval retry attempts for recurring snapshots #13454
Description
Activity
@KreVet-Ka , you say you reproduced. Did you force an error?
@KreVet-Ka , you say you reproduced. Did you force an error?
I first encountered this issue on the production infrastructure, and then successfully reproduced it on the test environment.
Steps to reproduce:- Reduce the available free space on the secondary storage so that it becomes less than the space required to create a snapshot.
- Create a recurring snapshot task for a disk.
In this case, each snapshot creation attempt fails with the following error:
“Error while creating entity for allocating snapshot”
However, after the failure, the system enters an infinite loop of snapshot creation attempts, with every subsequent attempt also failing with the same error.Reacted by dahn@KreVet-Ka , any chance to review/test #13743?
@KreVet-Ka , any chance to review/test #13743?
I’ll try, but I can’t promise it will work, since I’ve never built CloudStack myself.
Reacted by dahnI tried to build and run this PR locally on Ubuntu 24.04 (all-in-one: management + agent on the same host).
Build and packaging succeeded, packages installed fine, but Management Server fails to start.ERROR:
2026-08-03 08:37:45,678 INFO [c.c.u.c.ComponentContext] (main:[]) (logid:) Setup Spring Application context 2026-08-03 08:37:45,681 DEBUG [c.c.u.d.DbProperties] (main:[]) (logid:) DB properties were already loaded 2026-08-03 08:37:45,683 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2026-08-03 08:37:45,755 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2026-08-03 08:37:45,795 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2026-08-03 08:37:45,817 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2026-08-03 08:37:45,836 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2026-08-03 08:37:45,848 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2026-08-03 08:37:45,859 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2026-08-03 08:37:45,872 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2026-08-03 08:37:45,879 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2026-08-03 08:37:45,915 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2026-08-03 08:37:45,923 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2026-08-03 08:37:45,949 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2026-08-03 08:37:45,964 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2026-08-03 08:37:45,977 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2026-08-03 08:37:45,985 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2026-08-03 08:37:45,997 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2026-08-03 08:37:46,471 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from DataCenterIpAddressDaoImpl 2026-08-03 08:37:47,069 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2026-08-03 08:37:47,081 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from RoleManagerImpl 2026-08-03 08:37:47,425 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from HostTagsDaoImpl 2026-08-03 08:37:47,546 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ManagementServiceConfigurationImpl 2026-08-03 08:37:47,570 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from HostDaoImpl 2026-08-03 08:37:49,901 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2026-08-03 08:37:49,923 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [VpcVirtualRouterElement] in [Site2 Site Vpn Service Providers Registry] 2026-08-03 08:37:49,924 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from Site2SiteVpnManagerImpl 2026-08-03 08:37:50,436 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2026-08-03 08:37:50,661 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ClvmPoolManager 2026-08-03 08:37:50,945 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2026-08-03 08:37:50,962 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2026-08-03 08:37:50,971 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2026-08-03 08:37:51,015 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from VMSnapshotManagerImpl 2026-08-03 08:37:51,036 INFO [c.c.s.GsonHelper] (main:[]) (logid:) Default Builder inited. 2026-08-03 08:37:51,446 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from IndirectAgentLBServiceImpl 2026-08-03 08:37:51,487 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from UserDataManagerImpl 2026-08-03 08:37:51,507 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from PremiumSecondaryStorageManagerImpl 2026-08-03 08:37:51,666 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2026-08-03 08:37:51,688 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ManagementServerImpl 2026-08-03 08:37:51,857 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2026-08-03 08:37:51,863 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from KMSManagerImpl 2026-08-03 08:37:51,969 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from VolumeApiServiceImpl 2026-08-03 08:37:52,020 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2026-08-03 08:37:52,026 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from BackupManagerImpl 2026-08-03 08:37:52,085 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry 2026-08-03 08:37:52,116 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from SnapshotManagerImpl 2026-08-03 08:37:52,212 ERROR [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Failed to load module [core] due to: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ipDeployersRegistry' defined in URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.23.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-registry-core-context.xml]: Cannot resolve reference to bean 'VpcVirtualRouter' while setting bean property 'preRegistered' with key [0]; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'VpcVirtualRouter': Unsatisfied dependency expressed through field '_networkMdl'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'networkModelImpl': Unsatisfied dependency expressed through field '_accountMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'accountManagerImpl': Unsatisfied dependency expressed through field '_templateDao'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'VMTemplateDaoImpl': Unsatisfied dependency expressed through field '_templateDataStoreDao'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'templateDataStoreDaoImpl': Unsatisfied dependency expressed through field '_storeMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'dataStoreProviderManager': Unsatisfied dependency expressed through field 'primaryDataStoreProviderMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'primaryDataStoreProviderMgr': Unsatisfied dependency expressed through field 'storageMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'storageManagerImpl': Unsatisfied dependency expressed through field '_agentMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'clusteredAgentManagerImpl': Unsatisfied dependency expressed through field 'caService'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'caManager': Unsatisfied dependency expressed through field 'networkOrchestrationService'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'networkOrchestrator': Unsatisfied dependency expressed through field '_alertMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'alertManagerImpl': Unsatisfied dependency expressed through field '_capacityMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'capacityManagerImpl': Unsatisfied dependency expressed through field '_templatePoolDao'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'vMTemplatePoolDaoImpl': Unsatisfied dependency expressed through field 'dataStoreManager'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'dataStoreManagerImpl': Unsatisfied dependency expressed through field 'imageDataStoreMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'imageStoreProviderMgr': Unsatisfied dependency expressed through field '_statsCollector'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'statsCollector': Unsatisfied dependency expressed through field '_userVmMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'userVmManagerImpl': Unsatisfied dependency expressed through field 'resourceScheduleManager'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'ResourceScheduleManagerImpl': Unsatisfied dependency expressed through field 'workerList'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'VMScheduleWorker': Unsatisfied dependency expressed through field 'asyncJobManager'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'asyncJobManagerImpl': Unsatisfied dependency expressed through field '_jobDispatchers'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'vmWorkJobDispatcher': Unsatisfied dependency expressed through field '_vmMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'clusteredVirtualMachineManagerImpl': Unsatisfied dependency expressed through field '_haMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'highAvailabilityManagerExtImpl': Unsatisfied dependency expressed through field 'consoleProxyManager'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'consoleProxyManagerImpl': Unsatisfied dependency expressed through field 'resourceManager'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'resourceManagerImpl': Unsatisfied dependency expressed through field 'deploymentManager'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'DeploymentPlanningManager': Unsatisfied dependency expressed through field '_affinityGroupService'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'AffinityGroupServiceImpl': Unsatisfied dependency expressed through field '_domainMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'domainManagerImpl': Unsatisfied dependency expressed through field '_projectMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'projectManagerImpl': Unsatisfied dependency expressed through field '_configMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'configurationManagerImpl': Unsatisfied dependency expressed through field '_networkSvc'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'networkServiceImpl': Unsatisfied dependency expressed through field '_rulesMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'rulesManagerImpl': Unsatisfied dependency expressed through field '_ipAddrMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'ipAddressManagerImpl': Unsatisfied dependency expressed through field '_lbMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'loadBalancingRulesManagerImpl': Unsatisfied dependency expressed through field '_firewallMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'firewallManagerImpl': Unsatisfied dependency expressed through field '_vpcMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'vpcManagerImpl': Unsatisfied dependency expressed through field '_resourceLimitMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'resourceLimitManagerImpl': Unsatisfied dependency expressed through field '_userVmJoinDao'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'userVmJoinDaoImpl': Unsatisfied dependency expressed through field 'extensionHelper'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'ExtensionsManager': Unsatisfied dependency expressed through field 'clusterManager'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'clusterManagerImpl': Unsatisfied dependency expressed through field 'reconcileCommandService'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'reconcileCommandServiceImpl': Unsatisfied dependency expressed through field 'volumeManager'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'volumeOrchestrator': Unsatisfied dependency expressed through field '_tmpltMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'templateManagerImpl': Unsatisfied dependency expressed through field '_tmplFactory'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'templateDataFactoryImpl': Unsatisfied dependency expressed through field 'directDownloadManager'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'directDownloadManager': Unsatisfied dependency expressed through field 'volService'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'snapshotManagerImpl' defined in URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.23.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-server-core-managers-context.xml]: Initialization of bean failed; nested exception is java.lang.NullPointerException: Cannot invoke "java.util.Set.add(Object)" because "currentConfigs" is nullMaybe I built it incorrectly?
@KreVet-Ka , usually these exceptions are DB schema related. check your logs to see if there were any upgrade errors. It looks like it is related to the configuration table.
If not, restore you DB and try again.
as for build related, It might be, for instance a spring configuration missing but unlikely.
The required feature described as a wish
We are facing a problem with recurring snapshots for volumes. When a recurring snapshot job fails, ACS keeps making endless retry attempts.
The user does not receive any notifications when a recurring snapshot job fails. Only repeated retry attempts to create the snapshot are observed.
In our test case, we reproduced the issue with an error:
It would be good to have the ability to configure:
To avoid an infinite snapshot creation loop that ends with an error. After the maximum number of retries is exhausted, a notification should be sent informing that the job has failed.
Related discussion: Recurring snapshots: How to limit retry attempts or configure interval in global settings? #13439