Skip to content

Add Custom hypervisor minimal changes - #7692

Merged
yadvr merged 8 commits into
apache:mainfrom
shapeblue:upstream-custom-hw-pr
Aug 16, 2023
Merged

Add Custom hypervisor minimal changes#7692
yadvr merged 8 commits into
apache:mainfrom
shapeblue:upstream-custom-hw-pr

Conversation

@nvazquez

@nvazquez nvazquez commented Jun 26, 2023

Copy link
Copy Markdown
Contributor

Description

Design document: https://cwiki.apache.org/confluence/display/CLOUDSTACK/%5BDRAFT%5D+Minimal+changes+to+allow+new+dynamic+hypervisor+type%3A+Custom+Hypervisor

This PR introduces the minimal changes to add a new hypervisor type (internally named Custom in the codebase, and configurable display name), allowing to write an external hypervisor plugin as a Custom Hypervisor to CloudStack

The custom hypervisor name is set by the setting: 'hypervisor.custom.display.name'. The new hypervisor type does not affect the behaviour of any CloudStack operation, it simply introduces a new hypervisor type into the system.

CloudStack does not have any means to dynamically add new hypervisor types. The hypervisor types are internally preset by an enum defined within the CloudStack codebase and unless a new version supports a new hypervisor it is not possible to add a host of a hypervisor that is not in part of the enum. It is possible to implement minimal changes in CloudStack to support a new hypervisor plugin that may be developed privately

This PR is an initial work on allowing new dynamic hypervisor types (adds a new element to the HypervisorType enum, but allows variable display name for the hypervisor)

Proposed Future work:

Replace the HypervisorType from a fixed enum to an extensible registry mechanism, registered from the hypervisor plugin

Feature Specifications

  • The new hypervisor type is internally named 'Custom' to the CloudStack services (management server and agent services, database records).
  • A new global setting ‘hypervisor.custom.display.name’ allows administrators to set the display name of the hypervisor type. The display name will be shown in the CloudStack UI and API.
    • In case the ‘hypervisor.list’ setting contains the display name of the new hypervisor type, the setting value is automatically updated after the ‘hypervisor.custom.display.name’ setting is updated.
  • The new Custom hypervisor type supports:
    • Direct downloads (the ability to download templates into primary storage from the hypervisor hosts without using secondary storage)
    • Local storage (use hypervisor hosts local storage as primary storage)
    • Template format: RAW format (the templates to be registered on the new hypervisor type must be in RAW format)
  • The UI is also extended to display the new hypervisor type and the supported features listed above.
  • The above are the minimal changes for CloudStack to support the new hypervisor type, which can be tested by integrating the plugin codebase with this feature.

Use cases

This PR allows the cloud administrators to test custom hypervisor plugins implementations in CloudStack and easily integrate it into CloudStack as a new hypervisor type ("Custom"), reducing the implementation to only the hypervisor supported specific storage/networking and the hypervisor resource to communicate with the management server.

  • CloudStack admin should be able to create a zone for the new custom hypervisor and add clusters, hosts into the zone with normal operations
  • CloudStack users should be able to execute normal VMs/volumes/network/storage operations on VMs/volumes running on the custom hypervisor hosts

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

Having set 'hypervisor.custom.display.name' = 'MyCustomHW', added an extra zone with the hypervisor type 'MyCustomHW' and added one host:

  • Listing hosts, one per zone. The hypervisor type is set as the display name of the custom hypervisor
Screenshot 2023-08-06 at 10 38 56
  • Listing hosts in database. The hypervisor type is set as Custom, despite the display name which is variable
mysql> select name, hypervisor_type from host where status = 'Up' and type = 'Routing' and removed is null;
+------------------------------------------+-----------------+
| name                                     | hypervisor_type |
+------------------------------------------+-----------------+
| ref-trl-4901-k-Mr8-nicolas-vazquez-kvm2  | KVM             |
| ref-trl-4901-k-Mr8-nicolas-vazquez-test1 | Custom          |
+------------------------------------------+-----------------+
2 rows in set (0.00 sec)

How Has This Been Tested?

@nvazquez

Copy link
Copy Markdown
Contributor Author

@blueorangutan package

@nvazquez

Copy link
Copy Markdown
Contributor Author

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@nvazquez a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6342

@yadvr yadvr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, didn't test it but this would allow anybody to write an external hypervisor plugin

@github-actions

Copy link
Copy Markdown

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

@nvazquez

Copy link
Copy Markdown
Contributor Author

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@nvazquez a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6351

@nvazquez

Copy link
Copy Markdown
Contributor Author

@blueorangutan test

@blueorangutan

Copy link
Copy Markdown

@nvazquez a [SF] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@shwstppr shwstppr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code change looks okay but I fear we have some checks in code based on hypervisor type such vm migration with storage, find hosts for migration, etc. Custom hypervisor plugin may struggle there or may not give the desired results.

@codecov

codecov Bot commented Jun 27, 2023

Copy link
Copy Markdown

Codecov Report

Merging #7692 (a5568b8) into main (eb31e3d) will increase coverage by 0.81%.
Report is 6 commits behind head on main.
The diff coverage is 7.35%.

@@             Coverage Diff              @@
##               main    #7692      +/-   ##
============================================
+ Coverage     13.47%   14.28%   +0.81%     
- Complexity     9423     9993     +570     
============================================
  Files          2747     2748       +1     
  Lines        258853   258944      +91     
  Branches      40311    40325      +14     
============================================
+ Hits          34873    36994    +2121     
+ Misses       219579   217152    -2427     
- Partials       4401     4798     +397     
Files Changed Coverage Δ
...in/java/com/cloud/storage/GuestOSHypervisorVO.java 42.30% <0.00%> (ø)
server/src/main/java/com/cloud/api/ApiDBUtils.java 0.16% <0.00%> (-0.01%) ⬇️
...src/main/java/com/cloud/api/ApiResponseHelper.java 3.78% <0.00%> (ø)
...m/cloud/api/query/dao/DomainRouterJoinDaoImpl.java 0.47% <0.00%> (ø)
.../java/com/cloud/api/query/dao/HostJoinDaoImpl.java 0.35% <0.00%> (-0.01%) ⬇️
...om/cloud/api/query/dao/StoragePoolJoinDaoImpl.java 0.52% <0.00%> (ø)
...ava/com/cloud/api/query/dao/UserVmJoinDaoImpl.java 3.18% <0.00%> (ø)
...ava/com/cloud/api/query/dao/VolumeJoinDaoImpl.java 5.71% <0.00%> (-0.07%) ⬇️
.../cloud/configuration/ConfigurationManagerImpl.java 16.04% <0.00%> (-0.04%) ⬇️
.../java/com/cloud/hypervisor/HypervisorGuruBase.java 9.92% <0.00%> (ø)
... and 7 more

... and 46 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian test result (tid-6883)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 36152 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7692-t6883-kvm-centos7.zip
Smoke tests completed. 93 look OK, 20 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
ContextSuite context=TestRouterDHCPHosts>:setup Error 0.00 test_router_dhcphosts.py
ContextSuite context=TestRouterDHCPOpts>:setup Error 0.00 test_router_dhcphosts.py
ContextSuite context=TestProjectSuspendActivate>:setup Error 8.36 test_projects.py
test_01_add_delete_kubernetes_supported_version Error 0.05 test_kubernetes_supported_versions.py
ContextSuite context=TestMetrics>:setup Error 0.00 test_metrics_api.py
ContextSuite context=TestListIdsParams>:setup Error 0.00 test_list_ids_parameter.py
ContextSuite context=TestRouterDns>:setup Error 0.00 test_router_dns.py
ContextSuite context=TestLoadBalance>:setup Error 0.00 test_loadbalance.py
ContextSuite context=TestRouterDnsService>:setup Error 0.00 test_router_dnsservice.py
ContextSuite context=TestRouterServices>:setup Error 0.00 test_routers.py
ContextSuite context=TestIsolatedNetworks>:setup Error 0.00 test_routers_network_ops.py
ContextSuite context=TestRedundantIsolateNetworks>:setup Error 0.00 test_routers_network_ops.py
ContextSuite context=TestRouterIpTablesPolicies>:setup Error 0.00 test_routers_iptables_default_policy.py
ContextSuite context=TestVPCIpTablesPolicies>:setup Error 0.00 test_routers_iptables_default_policy.py
ContextSuite context=TestCpuCapServiceOfferings>:setup Error 0.00 test_service_offerings.py
ContextSuite context=TestServiceOfferings>:setup Error 0.24 test_service_offerings.py
ContextSuite context=TestVMWareStoragePolicies>:setup Error 0.00 test_storage_policy.py
test_01_create_template Error 1.11 test_templates.py
test_CreateTemplateWithDuplicateName Error 1.13 test_templates.py
test_02_1_create_template_with_checksum_sha1_negative Error 0.12 test_templates.py
test_02_create_template_with_checksum_sha1 Error 0.10 test_templates.py
test_03_1_create_template_with_checksum_sha256_negative Error 0.09 test_templates.py
test_03_create_template_with_checksum_sha256 Error 0.09 test_templates.py
test_04_1_create_template_with_checksum_md5_negative Error 0.09 test_templates.py
test_04_create_template_with_checksum_md5 Error 0.12 test_templates.py
test_05_create_template_with_no_checksum Error 0.08 test_templates.py
ContextSuite context=TestTemplates>:setup Error 259.76 test_templates.py
test_01_migrate_VM_and_root_volume Error 80.80 test_vm_life_cycle.py
test_02_migrate_VM_with_two_data_disks Error 52.38 test_vm_life_cycle.py
test_10_attachAndDetach_iso Error 0.07 test_vm_life_cycle.py
test_06_download_detached_volume Error 301.78 test_volumes.py
test_13_migrate_volume_and_change_offering Error 127.86 test_volumes.py
ContextSuite context=TestIpv6Vpc>:setup Error 0.00 test_vpc_ipv6.py
ContextSuite context=TestVPCRedundancy>:setup Error 0.00 test_vpc_redundant.py
ContextSuite context=TestVPCNics>:setup Error 0.00 test_vpc_router_nics.py
ContextSuite context=TestRVPCSite2SiteVpn>:setup Error 0.00 test_vpc_vpn.py
ContextSuite context=TestVPCSite2SiteVPNMultipleOptions>:setup Error 0.00 test_vpc_vpn.py
ContextSuite context=TestVpcRemoteAccessVpn>:setup Error 0.00 test_vpc_vpn.py
ContextSuite context=TestVpcSite2SiteVpn>:setup Error 0.00 test_vpc_vpn.py

@borisstoyanov

Copy link
Copy Markdown
Contributor

@blueorangutan test

@blueorangutan

Copy link
Copy Markdown

@borisstoyanov a [SF] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6659

@nvazquez

nvazquez commented Aug 5, 2023

Copy link
Copy Markdown
Contributor Author

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@nvazquez a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6673

@nvazquez

nvazquez commented Aug 5, 2023

Copy link
Copy Markdown
Contributor Author

@blueorangutan test matrix

@blueorangutan

Copy link
Copy Markdown

@nvazquez a [SF] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian Build Failed (tid-7292)

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian test result (tid-7290)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 40101 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7692-t7290-xenserver-71.zip
Smoke tests completed. 113 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian test result (tid-7291)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 74108 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7692-t7291-vmware-67u3.zip
Smoke tests completed. 109 look OK, 4 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
test_02_upgrade_kubernetes_cluster Failure 591.23 test_kubernetes_clusters.py
test_list_system_vms_metrics_history Failure 0.23 test_metrics_api.py
test_list_vms_metrics_admin Error 3607.24 test_metrics_api.py
test_list_vms_metrics_history Error 19.01 test_metrics_api.py
test_list_volumes_metrics_history Error 3607.11 test_metrics_api.py
test_01_deploy_vm_on_specific_host Error 11.52 test_vm_deployment_planner.py
test_02_deploy_vm_on_specific_cluster Error 3602.61 test_vm_deployment_planner.py
test_03_deploy_vm_on_specific_pod Error 4.40 test_vm_deployment_planner.py
test_04_deploy_vm_on_host_override_pod_and_cluster Error 2.36 test_vm_deployment_planner.py
test_05_deploy_vm_on_cluster_override_pod Error 2.31 test_vm_deployment_planner.py
test_09_expunge_vm Failure 424.66 test_vm_life_cycle.py

@nvazquez

nvazquez commented Aug 6, 2023

Copy link
Copy Markdown
Contributor Author

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@nvazquez a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✖️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6680

@nvazquez

nvazquez commented Aug 7, 2023

Copy link
Copy Markdown
Contributor Author

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@nvazquez a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6691

@nvazquez

nvazquez commented Aug 7, 2023

Copy link
Copy Markdown
Contributor Author

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@nvazquez a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6693

@DaanHoogland DaanHoogland left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the code looks good to me. I really don't like the design of it though. This hack of adding a custom enum type for HypervisorType is ugle. I would really hope this is the start of a longer term project to make hypervisors truly plugable.

@borisstoyanov borisstoyanov left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yadvr
yadvr marked this pull request as ready for review August 16, 2023 15:21

@yadvr yadvr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, relying on @borisstoyanov 's testing - this would enable hypervisor plugin developers to be able to build/ship custom plugins outside of CloudStack release cycle, much like it's possible with storage plugins. While the implementation can be improved to allow a more pluggable model, this is good enough for the first iteration.

@yadvr
yadvr merged commit 8b5ba13 into apache:main Aug 16, 2023
@yadvr
yadvr deleted the upstream-custom-hw-pr branch August 16, 2023 15:26
shwstppr pushed a commit to shapeblue/cloudstack that referenced this pull request Oct 12, 2023
Design document: https://cwiki.apache.org/confluence/display/CLOUDSTACK/%5BDRAFT%5D+Minimal+changes+to+allow+new+dynamic+hypervisor+type%3A+Custom+Hypervisor

This PR introduces the minimal changes to add a new hypervisor type (internally named Custom in the codebase, and configurable display name), allowing to write an external hypervisor plugin as a Custom Hypervisor to CloudStack

The custom hypervisor name is set by the setting: 'hypervisor.custom.display.name'. The new hypervisor type does not affect the behaviour of any CloudStack operation, it simply introduces a new hypervisor type into the system.

CloudStack does not have any means to dynamically add new hypervisor types. The hypervisor types are internally preset by an enum defined within the CloudStack codebase and unless a new version supports a new hypervisor it is not possible to add a host of a hypervisor that is not in part of the enum. It is possible to implement minimal changes in CloudStack to support a new hypervisor plugin that may be developed privately

This PR is an initial work on allowing new dynamic hypervisor types (adds a new element to the HypervisorType enum, but allows variable display name for the hypervisor)

Replace the HypervisorType from a fixed enum to an extensible registry mechanism, registered from the hypervisor plugin

- The new hypervisor type is internally named 'Custom' to the CloudStack services (management server and agent services, database records).
- A new global setting ‘hypervisor.custom.display.name’ allows administrators to set the display name of the hypervisor type. The display name will be shown in the CloudStack UI and API.
   - In case the ‘hypervisor.list’ setting contains the display name of the new hypervisor type, the setting value is automatically updated after the ‘hypervisor.custom.display.name’ setting is updated.
- The new Custom hypervisor type supports:
   - Direct downloads (the ability to download templates into primary storage from the hypervisor hosts without using secondary storage)
   - Local storage (use hypervisor hosts local storage as primary storage)
   - Template format: RAW format (the templates to be registered on the new hypervisor type must be in RAW format)
- The UI is also extended to display the new hypervisor type and the supported features listed above.
- The above are the minimal changes for CloudStack to support the new hypervisor type, which can be tested by integrating the plugin codebase with this feature.

This PR allows the cloud administrators to test custom hypervisor plugins implementations in CloudStack and easily integrate it into CloudStack as a new hypervisor type ("Custom"), reducing the implementation to only the hypervisor supported specific storage/networking and the hypervisor resource to communicate with the management server.

- CloudStack admin should be able to create a zone for the new custom hypervisor and add clusters, hosts into the zone with normal operations
- CloudStack users should be able to execute normal VMs/volumes/network/storage operations on VMs/volumes running on the custom hypervisor hosts

(cherry picked from commit 8b5ba13)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants