extensions: add sync, download functionalities by shwstppr · Pull Request #11814 · apache/cloudstack

shwstppr · 2025-10-09T09:40:16Z

Description

This change introduces support for:

synchronizing extension files across management servers in a clustered environment.
download extension files as an archive (from a desired management server in a clustered environment). If needed (cases when MS is not directly accessible), download can be made to work via SSVM and secondary storage.

Key changes:

Adds new syncExtension API to trigger synchronization for a selected extension.
Implements util to create .tgz archive of the extension directory or selected files for sync.
For sync, sends DownloadAndSyncExtensionFilesCommand to peer management servers. Handles archive download, staging, and extraction on the receiver side.
Adds new downloadExtension API to allow downloading extension files as an archive.
Implements util to create .zip archive of the extension directory for download.
For forcing download via secondary storage, global configuration - extension.share.download.use.secondary.storage can be used and set to true.
Generates signed share URL with HMAC signature and expiry.
URL expiry and file cleanup can be controlled via global configuration - extension.share.link.validity.interval
Adds Sync Extension and Download Extension actions in the UI Extensions view.
Updates events, logging, and cleanup of temporary share files.
Refactors all filesystem-related code to the extensions framework layer.
Checksum for extensions is now calculated and compared for all files in the extension directory.

Documentation PR: apache/cloudstack-documentation#624

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

For Sync:

(localcloud) 🐱 > sync extension id=3cf82f18-89c5-4acc-8f46-8d42da139f30 sourcemanagementserverid=0d8d73fa-9241-49aa-8d7e-7642f143ce66
{
  "success": true
}
(localcloud) 🐱 > sync extension id=3cf82f18-89c5-4acc-8f46-8d42da139f30
💩 Missing required parameters:  sourcemanagementserverid
(localcloud) 🐱 > sync extension -h
syncExtension: To sync the extension files from one management server to other management server(s)
This API is asynchronous.
Required params: id, sourcemanagementserverid, 
API Params               Type     Description
==========               ====     ===========
files                    list     List of files to sync. Specify absolute o
                                  r relative paths. If not provided, all e
                                  xtension files will be synced.
id                       uuid     ID of the extension
sourcemanagementserverid uuid     ID of the management server from which fi
                                  les are to be synced
targetmanagementserverids list     the IDs of the management servers to whic
                                  h the extension files are to be synced. 
                                  If not specified, the files will be sync
                                  ed to all management servers

For Download:

(localcloud) 🐱 > download extension -h
downloadExtension: To download the extension files as an archive
This API is asynchronous.
Required params: id, 
API Params               Type     Description
==========               ====     ===========
id                       uuid     ID of the extension
managementserverid       uuid     ID of the management server from which fi
                                  les are to be downloaded
(localcloud) 🐱 > download extension id=3cf82f18-89c5-4acc-8f46-8d42da139f30
{
  "extension": {
    "id": "3cf82f18-89c5-4acc-8f46-8d42da139f30",
    "managementserverid": "0d8d73fa-9241-49aa-8d7e-7642f143ce66",
    "managementservername": "qa1-4.22-test-61f54c88-kvm-mgmt1",
    "name": "test",
    "url": "http://172.120.0.228:8080/share/extensions/test-1760334975686.zip?exp=1760338575"
  }
}
(localcloud) 🐱 > download extension id=3cf82f18-89c5-4acc-8f46-8d42da139f30 managementserverid=5ccb810f-4a31-4b8d-a36c-2ebb7e87cb26 
{
  "extension": {
    "id": "3cf82f18-89c5-4acc-8f46-8d42da139f30",
    "managementserverid": "5ccb810f-4a31-4b8d-a36c-2ebb7e87cb26",
    "managementservername": "422-test-mgmt2",
    "name": "test",
    "url": "http://172.120.0.83:8080/share/extensions/test-1760334994612.zip?exp=1760338594"
  }
}

How Has This Been Tested?

How did you try to break this feature and the system with this change?

codecov · 2025-10-09T09:48:28Z

Codecov Report

❌ Patch coverage is 62.03008% with 606 lines in your changes missing coverage. Please review.
✅ Project coverage is 18.07%. Comparing base (82bfa9f) to head (23360ac).

Files with missing lines	Patch %	Lines
...extensions/manager/ExtensionsShareManagerImpl.java	76.47%	66 Missing and 50 partials ⚠️
...sions/manager/ExtensionsFilesystemManagerImpl.java	65.35%	64 Missing and 33 partials ⚠️
...work/extensions/manager/ExtensionsManagerImpl.java	73.14%	24 Missing and 34 partials ⚠️
...amework/extensions/ExtensionArchiveDataObject.java	12.50%	48 Missing and 1 partial ⚠️
...dstack/storage/to/DownloadableArchiveObjectTO.java	0.00%	48 Missing ⚠️
.../cloudstack/utils/server/ServerPropertiesUtil.java	0.00%	39 Missing ⚠️
...om/cloud/storage/download/DownloadMonitorImpl.java	0.00%	31 Missing ⚠️
...oudstack/storage/template/DownloadManagerImpl.java	0.00%	31 Missing ⚠️
...dstack/storage/image/BaseImageStoreDriverImpl.java	0.00%	28 Missing ⚠️
...pache/cloudstack/utils/filesystem/ArchiveUtil.java	69.04%	20 Missing and 6 partials ⚠️
... and 17 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #11814      +/-   ##
============================================
+ Coverage     17.95%   18.07%   +0.12%     
- Complexity    16522    16739     +217     
============================================
  Files          6022     6037      +15     
  Lines        541387   542682    +1295     
  Branches      66346    66482     +136     
============================================
+ Hits          97211    98105     +894     
- Misses       433210   433485     +275     
- Partials      10966    11092     +126

Flag	Coverage Δ
uitests	`3.52% <ø> (-0.01%)`	⬇️
unittests	`19.23% <62.03%> (+0.13%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

This change introduces support for synchronizing extension files across management servers in a clustered environment. - Adds new syncExtension API to trigger synchronization for a selected extension. - Implements service to create .tgz archive of the extension directory or selected files. - Generates signed share URL with HMAC signature and expiry. - Sends DownloadAndSyncExtensionFilesCommand to peer management servers. - Handles archive download, staging, and extraction on receiver side. - Adds Sync Extension action in the UI Extensions view. - Updates events, logging, and cleanup of temporary share files. Refactors all filesystem related code to extensions framework layer. Checksum for extensions is now calculated and compared for all files in the extension directory. Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

github-actions · 2025-10-17T13:28:55Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr · 2026-01-31T19:10:48Z

@blueorangutan test

blueorangutan · 2026-01-31T19:12:03Z

@shwstppr a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2026-02-01T10:55:51Z

[SF] Trillian test result (tid-15347)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 53990 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11814-t15347-kvm-ol8.zip
Smoke tests completed. 147 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestListIdsParams>:teardown	`Error`	1.12	test_list_ids_parameter.py
test_01_snapshot_root_disk	`Error`	5.03	test_snapshots.py
test_02_list_snapshots_with_removed_data_store	`Error`	47.81	test_snapshots.py
test_02_list_snapshots_with_removed_data_store	`Error`	47.82	test_snapshots.py
ContextSuite context=TestSnapshotStandaloneBackup>:teardown	`Error`	31.70	test_snapshots.py
test_01_snapshot_usage	`Error`	25.73	test_usage.py
test_01_vpn_usage	`Error`	1.10	test_usage.py

harikrishna-patnala

LGTM. Tested the sync, downloads operations and they are working fine as per the configurations introduced.

harikrishna-patnala · 2026-02-05T07:38:45Z

@shwstppr please check if we need to add these settings in the existing documentation.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr · 2026-02-06T09:21:23Z

@blueorangutan package

blueorangutan · 2026-02-06T09:22:04Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-02-06T10:07:45Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16727

For: apache/cloudstack#11814 Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr · 2026-02-12T04:53:07Z

@harikrishna-patnala documentation added here, apache/cloudstack-documentation#624

github-actions · 2026-02-17T11:45:19Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

github-actions · 2026-04-14T17:28:12Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

sudo87

Left a small comment, but overall this looks good to me.

shwstppr · 2026-04-16T06:08:46Z

@blueorangutan package

blueorangutan · 2026-04-16T06:10:04Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-04-16T06:49:38Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✖️ debian ✔️ suse15. SL-JID 17518

shwstppr · 2026-04-17T11:51:30Z

@blueorangutan test

blueorangutan · 2026-04-17T11:52:04Z

@shwstppr a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

boring-cyborg bot added component:api component:management-server component:UI labels Oct 9, 2025

shwstppr force-pushed the feature-sync-ext branch from af72efe to d98ff4f Compare October 9, 2025 10:06

shwstppr added 11 commits October 10, 2025 10:33

Merge remote-tracking branch 'apache/main' into feature-sync-ext

d7d9a4a

more tests

4a535bd

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

missing file

0337e34

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

fix line endings

c91a970

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

Merge branch 'main' into feature-sync-ext

d384fb2

do not use mockConstruction

ecbdcd6

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

changes

2c70e03

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

changes for download extension

2be9d96

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

fix lint and rat check

40b013d

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

add more tests

2da3004

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

fix lint

fdfcfe6

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr changed the title ~~extensions: add sync functionality~~ extensions: add sync, download functionalities Oct 13, 2025

apache deleted a comment from blueorangutan Oct 13, 2025

handle when share context is disabled

942f36e

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

apache deleted a comment from blueorangutan Oct 14, 2025

github-actions bot added the status:has-conflicts label Oct 17, 2025

Merge remote-tracking branch 'apache/main' into feature-sync-ext

f923de8

github-actions bot removed the status:has-conflicts label Oct 26, 2025

changes for download via ssvm/secondary store

5525242

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

boring-cyborg bot added the component:storage label Oct 29, 2025

allow cleanup of archives on sec store

865c6fd

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

harikrishna-patnala approved these changes Feb 5, 2026

View reviewed changes

harikrishna-patnala added the status:tested label Feb 5, 2026

shwstppr added 2 commits February 6, 2026 14:13

refactor

6941097

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

address copilot comments

6519e55

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr mentioned this pull request Feb 12, 2026

extensions: sync and download apache/cloudstack-documentation#624

Open

shwstppr added a commit to shwstppr/cloudstack-documentation that referenced this pull request Feb 12, 2026

extensions: sync and download

0d9da1a

For: apache/cloudstack#11814 Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

github-actions bot added the status:has-conflicts label Feb 17, 2026

Merge remote-tracking branch 'apache/main' into feature-sync-ext

e895355

github-actions bot added status:has-conflicts and removed status:has-conflicts labels Apr 13, 2026

Merge remote-tracking branch 'apache/main' into feature-sync-ext

49d9b76

shwstppr removed the status:has-conflicts label Apr 14, 2026

sudo87 reviewed Apr 16, 2026

View reviewed changes

Comment thread api/src/main/java/com/cloud/event/EventTypes.java

sudo87 approved these changes Apr 16, 2026

View reviewed changes

address comment

23360ac

Conversation

shwstppr commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

codecov bot commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Oct 17, 2025

Uh oh!

shwstppr commented Jan 31, 2026

Uh oh!

blueorangutan commented Jan 31, 2026

Uh oh!

blueorangutan commented Feb 1, 2026

Uh oh!

harikrishna-patnala left a comment

Choose a reason for hiding this comment

Uh oh!

harikrishna-patnala commented Feb 5, 2026

Uh oh!

shwstppr commented Feb 6, 2026

Uh oh!

blueorangutan commented Feb 6, 2026

Uh oh!

blueorangutan commented Feb 6, 2026

Uh oh!

shwstppr commented Feb 12, 2026

Uh oh!

github-actions bot commented Feb 17, 2026

Uh oh!

github-actions bot commented Apr 14, 2026

Uh oh!

Uh oh!

sudo87 left a comment

Choose a reason for hiding this comment

Uh oh!

shwstppr commented Apr 16, 2026

Uh oh!

blueorangutan commented Apr 16, 2026

Uh oh!

blueorangutan commented Apr 16, 2026

Uh oh!

shwstppr commented Apr 17, 2026

Uh oh!

blueorangutan commented Apr 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

shwstppr commented Oct 9, 2025 •

edited

Loading

codecov bot commented Oct 9, 2025 •

edited

Loading