Your customers will be able to create as many servers as they need, manage such essential aspects like backups or firewall, and access noVNC, SPICE, or Xterm.js console.
The module supports Cloud-Init for the KVM virtualization type, and features Proxmox Addon allowing you to easily manage servers, IP addresses and clusters.

We will guide you step by step through the whole installation and configuration process.

In the most recent versions of the module, you will find only one package that supports PHP 7.1 and later.

The first one that supports PHP 5.6 up to PHP 7.0, and the second one addressed to PHP 7.1 up to PHP 7.3.
Note: Keep in mind that PHP versions 5.6 up to 7.0 are no longer officially supported and their security issues are not fixed or released any more. Find more info here.

The content of the package to upload should look like this.

Note: If you wish to allow noVNC consoles, you need to configure a product using the PVE realm and set user permissions to 'PVEVMUser.'
Important: Open Folder UploadOnProxmoxServer/5.2-10 and upload content on all your proxmox servers. Scripts must be updated on proxmox server v2.6.0 to allow Xterm.js console work.

This file is located in 'modules/servers/proxmoxCloud/license_RENAME.php'. Rename it from 'license_RENAME.php' to 'license.php'.

Enter your license key between quotation marks as presented on the following screen.
You can find your license key in our client area → 'My Products'.

This folder is available at: your_whmcs/modules/addons/proxmoxAddon/

To do so, log in into your WHMCS and press 'Setup' → 'Addon Modules'. Find 'Proxmox Addon' and press 'Activate'.

Note: The Proxmox addon version might not match the current version of the provisioning module itself. This is completely normal due to the specific structure of the product.

To do so, click on the 'Configure' button. Tick 'Full Administrator' and press 'Save Changes'.

Go to 'Setup' → 'Products/Services' → 'Servers' and press 'Add New Server'.

Choose 'ProxmoxCloud' from a dropdown menu and press 'Save Changes'.
The last step is choosing authentication from the 'Authentication' dropdown menu.
Note: Authentication depends on the realm used by the user on a Proxmox server.

You can check connection with the Proxmox server. To do so, simply press 'Test Connection' as shown on the screen below.

Click on 'Create a New Group'.

To create a product, click on 'Create a New Product'.

'Product Type' can be set to 'Server/VPS' but nameservers will need to be added later.
If you do not want to fill out nameservers, please select 'Shared Account' .

Next, click on 'Save Changes'.

Available virtualization types:

• KVM
  
• LXC
  

You can also choose how often the bandwidth usage should be calculated.
Afterwards, press 'Save Changes'.
Note: If you choose 'Auto-Node' as 'Default Node', new virtual servers will be created on the least used node.
Note 2: An advanced tool is available for you to automatically manage a node selection with the best count of resources. Read more about Load Balancer.

From dropdown menus, choose 'KVM OS Type', 'Default Storage', 'Disk Type' and 'CD/DVD-ROM Type'.
Choose ISO images available for your clients by marking them in 'ISO Image'.

Set the network through selecting from the 'Network Mode', 'Network Model', and 'Bridge' dropdown menus.
If you are going to allow your clients to manage their private networks, you have to select 'Private Bridge' and set up VLAN TAG Range.
Type in your VM description in 'Container Description', set the backup location in the 'Storage' dropdown menu and click 'Save Changes'.
Important: Keep in mind that the root password is generated automatically for KVM virtualization upon its creation and reset.
'If you want to set your own password on KVM virtualization, use the Cloud-Init option.

From dropdown menus, choose 'Default Storage', 'Template' and 'Bridge'.
Enter your virtual server description in 'Container Description' and press 'Save Changes'.

Remember! This virtualization type is supported only by Proxmox VE 4.0. and later.
If you are using the previous version of a Proxmox server, you will not be able to start this virtualization.

• Select 'Private Bridge'
• Select 'Private Network Model'
• Enter ranges for VLAN TAG
  

Note: Configure these three options for the KVM and LXC virtualization.
If you skip this step, clients will not be allowed to add private networks.

Define backup settings in the 'Backups' section through selecting values from the 'Storage' and 'Backups Routing' dropdown menus.
Backups routing automatically removes the oldest backup when a new backup is created and the established backup limit is reached.
Additionally, you can limit the time period for storing backups by providing the number of days into the 'Store The Backup For N Days' field.
Furthermore, you can define the limit of firewall rules and the interface to apply the rules to.
Finish the configuration in the 'Module Settings' tab through pressing 'Save Changes'.

Press 'Generate default' next to 'Custom Fields'.
Static Resources: Fill in the fields in located in the 'Default Configuration' section with desired values.
Configurable Resources: Click on 'Generate Default' next to 'Configurable Options'.
Press 'Save Changes' at the bottom of the page.
Note: While generating 'Configurable Options', it is advisable to delete the ones you do not use, e.g. the swap option or IPv6 for the KVM virtualization.

php -q /your_whmcs/modules/servers/proxmoxCloud/cron/cron.php

Do not forget to replace 'your_whmcs' with your WHMCS root location.

Please note that there are separate cron jobs dedicated to Proxmox VPS and Proxmox Cloud modules - these jobs are the most important to assure fluent work of the modules, suggested frequency is 5 minutes.
If you are using only Proxmox VPS, there is no need to set up the second one and the other way round.
The other ones are responsible for:

• Users synchronization - run only once
• VM recovery synchronization - optionally run every 24 hours
• VM migration synchronization - optionally run every 24 hours or just after the VM migration
• Tasks synchronization - optionally run every one hour

In order to create a new product, you have to add 'Dedicated IP ' to the 'IP Addresses' list in 'Proxmox' addon module.
To do so, go to 'IP Management' and press 'New IP Addresses'.
Note: You can add both the IPv4 and IPv6 addresses.

Next, choose a server from the 'Server' dropdown menu and define which virtualization it should be applied to through selecting it from the 'Virtualization' dropdown menu.
Afterward, press 'Confirm'.

If you used the KVM virtualization type, the client area of a successfully created product should look similar to the one presented on the screen below.

Before you start we recommend to learn the details about cloud-init configuration on your Proxmox server - click here for more details.
Once you get familiar with Proxmox cloud-init proceed to Cloud-Init group of your KVM configuration where you can enable it and set the following:

• Allow To Set Username - if enabled the client can set its own username during VM creation. Otherwise it will remain unchanged and be set by default as in the template or set from the 'Default User' field if provided.
• Allow To Set Password - if enabled the client can set its own password during VM creation Otherwise it will remain unchanged and be set by default as in the template.
• Allow To Set SSH Public Key - if enabled the client can upload its own public SSH key in OpenSSH format during VM creation. This will then allow client to log in to VM using their private key instead of password.
• Default User - provide the username that will be used for VM instead of using the 'Service Username' field or image's configured default user. Otherwise leave empty.
• Allow To Set DNS Domain - if enabled the client can provide its own DNS domain for a container. During VM creation, the provided setting will be automatically used if neither a search domain nor a nameserver is set.
• Allow To Set Nameservers - if enabled the client can set its own nameservers for the VM. Otherwise they will remain unchanged and be set by default as in the template.

Additionally you can use the following commands that can be useful for debugging.

• Re-run cloud init scripts

rm -rf /var/lib/cloud/sem/* /var/lib/cloud/instance /var/lib/cloud/instances/* &&
cloud-init init  &&
cloud-init modules -m final

• Logs

cat /var/log/cloud-init.log
cat /var/log/cloud-init-output.log

At the same time, you can monitor and handle each product from your WHMCS admin area.
To view VM details(1), migrate VM(2), update VMs node(3) or delete VM(4), press a specific button.
To view details of a virtual server created by your client, simply press the 'Load Status' button.

Migration is useful when you want to request migration from the WHMCS admin area, while editing node is practical for updating details after migrating the virtual machine from the Proxmox panel.

RAM details:

Assigned - RAM assigned to the server (hosting services only)

Total - full server RAM

Free - difference between Total Server RAM and Assigned RAM

Suspended - number of suspended accounts per size in MB E.g.: (1/500 MB) when there is only one suspended account (occupying 500MB of that account) assigned to Proxmox VPS server.

Press 'More Details' icon and you will get extended information on the selected server. See below for more details.

To view information about a VM, simply press the more info icon.

In order to convert a VM, press 'Create Template' button.

Fill out 'Description' field and press 'confirm'.
Important: Selected VM will be destroyed in the process.

Note: VM ID range allows you to sort created VMs depending on the server they were created on..

Run the 'VM recovery synchronization' cron job to create and store full backups of VMs configuration, you can find it at point no 26 of the configuration instructions.
Use the 'Export to File' button to save information in a TXT copy file.

Run the 'Tasks synchronization' cron job to synchronize the tasks and view all tasks from the server, you can find it at point no 26 of the configuration instructions.

• private/public address
• servers or nodes that will use them
• virtualization types (different VMs will use different IP pools)

By activating the IP Manager submodule, settings concerning IP addresses such as 'Configurable Options' and 'Default Options' will be ignored by Proxmox VPS.
Note: This integration works for LXC and KVM virtualization types.

1. Proceed to the IP Manager For WHMCS → Subnets tab in order to prepare IP addresses subnet for Proxmox VPS For WHMCS.
2. Edit the previously created IP addresses subnet and create a custom field 'MAC Address' with a desired MAC address at each IP address in the subnet.
Note: Setting up MAC address is optional and can be skipped.
More details about creating custom fields can be found here.

Note: Multiple IP addresses subnets can be assigned to a single configuration.
4. Now, link the configuration with Proxmox VPS product via a product, a server or a configurable option.
On the following screen we have linked them via the configurable option assigned to the Proxmox VPS product.

Press 'Activate' next to the 'proxmoxVPS' module type as shown on the following screen.

Press 'Configure' , choose a configuration and press 'Save Changes'.

Look above for the detailed instruction on how to proceed, follow steps 1-6 to create the new custom field.
The only thing you need to change is the custom field name, which should be 'Tag' in this case.

Set the weight of the resource that will prioritize its selection when sorting the Nodes according to the free resources.
Note: The lowest possible weight is 1. You cannot set for example 0,5.

Define the behavior of the tool:

• On Upgrade - select action to perform:
  • None - if selected, no action will be triggered
  • Block - the upgrade process will blocked as there are insufficient resources on the node
  • Migrate - the upgrade process will be performed after previous migration of the VM to a server with sufficient free space
• Shutdown VM on Upgrade
• Stop VM If Shutdown Fails

Let's take exemplary weights for each resource just like presented on the screen above.

VMs Number Weight: 1,5

CPU Weight: 2 

Disk Weight: 4

RAM Weight: 2,5 

Disk Weight with the highest weight which means it has the highest priority and VMs Number Weight as the least crucial factor on a given node.

In our situation, there are four nodes created, named A, B, C, D with current resources as shown below:

Node A (Free Ram: 60%, Free CPU: 20%, Free Disk Space: 50%, Number of VMs: 5) 

Node B (Free Ram: 30%, Free CPU: 70%, Free Disk Space: 40%, Number of VMs: 10) 

Node C (Free Ram: 90%, Free CPU: 50%, Free Disk Space: 20%, Number of VMs: 20) 

Node D (Free Ram: 80%, Free CPU: 60%, Free Disk Space: 70%, Number of VMs: 15)

The next step is sorting the nodes according to each resource, by assigning them corresponding priorities from 1 to 4.
The higher the priority, the more free resources available.

Sorting nodes is based on the following calculation method:

Final Node Priority = (RAM Priority x RAM Weight) + (CPU Priority x CPU Weight) + (Free Disk Priority x Disk Weight) + (No. VMs  Priority x VMs Weight)

The results for nodes A, B, C and D (with weights and sorting priority) are following:

Node A: (2 x 2,5) + ( 1 x 2 ) + ( 3 x 4 ) + ( 4 x 1,5 ) = 25
Node B: (1 x 2,5) + ( 4 x 2 ) + ( 2 x 4 ) + ( 3 x 1,5 ) = 23
Node C: (4 x 2,5) + ( 2 x 2 ) + ( 1 x 4 ) + ( 1 x 1,5 ) = 19,5
Node D: (3 x 2,5) + ( 3 x 2 ) + ( 4 x 4 ) + ( 2 x 1,5 ) = 32,5

As you can see, Node D has got the highest priority value with a result: 32,5.
It will be selected by Load Balancer to create a new VM there, to migrate or upgrade an existing VM if its current node does not have sufficient resources.

The above described calculations are performed every time, a 'Create' or 'Upgrade' actions are executed based on the current usage of resources on available nodes.

Important: In case a given client already has an existing VM on a node, this very node will NOT be taken into account during the prioritization of resources process.
This is due to security measures to make sure, that in case of failure of this node, the client will still have access to the other VM server, created on another, working node.
However, if this client has VMs on every node already, all of them will be taken into account during the selection prioritization process again.

If you decide to use Load Balancer tool, define the resources weights to adjust their prioritization of nodes according to your needs.

They can also view resources used and manage existing virtual servers.

Instead of using the sliders you can also enter resources values into textboxes.
Confirm creation through pressing 'Add New Virtual Server'.
Note! Data to be filled in may differ according to which type LXC or KVM you have chosen.

You can find information about managing your virtual servers in next section.

Please keep in mind that this option must be enabled by administrator in your product's 'Module Settings' previously and a private IP address must be added in the Proxmox Addon.
Otherwise you will not be able to add any private network successfully.

OS templates do not consume any resources such as CPU sockets, memory etc.
Important! When you convert server into a template, it is consumed in the process! There is no possibility to revert it back.

As you can see, control panel contains useful options needed for managing VPS.
The module allows your customers to easily boot, stop, reboot and shutdown their VM through pressing a specific button.
Clients can also upgrade server and access VNC, SPICE and noVNC consoles.
Any additional tools are described in details in below sections.

Note: Your client needs to download and install client side SPICE application in order to connect with VM.

You must update the scripts on Proxmox v2.6.0. To do so open folder: UploadOnProxmoxServer/5.2-10 and upload content on all your proxmox servers.

Afterwards, alter options you desire and confirm through pressing 'Save Changes' button.

You can provide your clients to either select a new OS from a provided KVM templates to automatically system replacement, or let them choose the ISO image for manual system installation.

Note: The ISO images can only be used for the KVM virtualization type, while VM templates can be used either for KVM or LXC.
Additionally the new OS password for the KVM OS reinstallation will be set based on the previous service password while for the LXC user can define it before the new installation.

Afterwards, press 'Add' button.

To add a new backup job press 'New Job' as shown on the following screen.

Next, press 'Add' button. You may also enable email notifications here.

They can also switch the timeline of the graphs by choosing it from a dropdown menu pointed on the following screen.

Additionally, each access to the VM through SPICE console is shown here.

Depending on the settings of the product (private network is used), they can also create/remove them. This feature is optional.

As you can see, you can easily modify and remove previously added rules and security groups.
To add a new firewall rule (1) or predefined security group (2) use the buttons marked on the following screen.

The first way (red) is by choosing macro from a 'Macro' dropdown field.
The second way (blue) consist of choosing protocol and filling source and destination ports out.
Additionally, you can enable rule and add a comment to it.
Press 'Add' to create a rule.

Additionally, you can enable security group and add a comment to it.
Click 'Add' to use the security group.

Note: This integration works for both LXC and KVM virtualization types.

1. Proceed to IP Manager For WHMCS → Subnets tab in order to prepare IP addresses subnet for Proxmox Cloud For WHMCS.
2. Edit a previously created IP addresses subnet and create custom field 'MAC Address' with desired MAC address at each IP address in the subnet.
More about creating custom fields can be found here.

Note: Multiple IP addresses subnets can be assigned to a single configuration.
4. Now, link the configuration with Proxmox Cloud product via product, server or configurable option.
On the following screen we have linked them via a configurable option assigned to the Proxmox Cloud product.

Press 'Activate' next to 'proxmoxCloud' module type as shown on the following screen.

Press 'Configure', choose the configuration and press 'Save Changes'.

To do so, press 'Import VM' as shown on the following screen.

Confirm importing through pressing 'Save Changes' button.

Custom hooks are placed in 'your_whmcs/modules/servers/proxmoxCloud/customhooks.php' .
You can find there all custom hooks and basic information about them.
A full list of available custom hooks:

proxmoxCloud_beforeCreateVM 
proxmoxCloud_afterCreateVM
proxmoxCloud_beforeTerminateVM
proxmoxCloud_afterTerminateVM
proxmoxCloud_beforeCreate
proxmoxCloud_afterCreate
proxmoxCloud_beforeTerminate
proxmoxCloud_afterTerminate
proxmoxCloud_afterSuspend
proxmoxCloud_afterUnsuspend
proxmoxCloud_afterIPAdditionByAdmin
proxmoxCloud_afterIPDeletionByAdmin
proxmoxCloud_afterIPAdditionByUpgrade

Note! If automatically generated configurable options do not seem to fulfill your clients needs, you can edit them using this guide.

Module allows you to charge your customers basing on the server resources used by them.

To achieve that you need our Proxmox Cloud Autoscaling extension with Advanced Billing For WHMCS product.
Only then, when you have such a combination of products your offer will be full.

Advanced Billing For WHMCS will bill your clients for the currently used servers and its extension, Proxmox Cloud Autoscaling, will adjust the size of Cloud product to the current needs.

When they are joined into one cluster, you can access all of them by connecting to any of them.
While auto-node creates new VM on the server(node) with the most free resources, server-node creates VM on the server(node) which IP address it has (WHMCS's server configuration).

$_LANG['node_X']='Your friendly name for node_X';

Keep in mind there is a quick and easy option in our client area that will let you upgrade the license of your module to an open source version at any time - and at a lower price!
To take advantage of it, simply use the License Upgrade button located on the product's page and a discounted invoice will be generated automatically.

Reason: User account does not have permission to access noVNC console.
Solution: Proceed to product's 'Module Settings' tab and check whether selected user permission grant right for noVNC console.
Afterwards, proceed to client's hosting and run 'Change User Role' on it.

Reason: Your certificate is not valid.
Quick Solution: Your client needs to add Proxmox address to certificate exception list.
Best Solution: Configure valid certificate.

Reason: You have not uploaded the additional files on the Proxmox server.
Solution: Open 'UploadOnProxmoxServer' and upload content of proper version on all Proxmox servers into the '/' path.

Reason: You have not updated php.ini file
Solution: After SSH2 PECL installation be sure that you have updated php.ini files for CLI and webserver.

Solution: Turn off the Xterms.js console in the product configuration.

Reason: User is other than "root" or server authentication is set to Proxmox VE
Solution: In server configuration, set the root details and select PAM as the authentication type.