Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ By default, short-term forecasts cover the next five minutes.

## Forecast Generation

Forecasts rely on OpenNebula’s built-in monitoring system ([The OpenNebula Monitoring System]({{% relref "product/cloud_system_administration/scheduler/configuration/#monitor-alert-configuration" %}})). Periodically, monitoring probes analyze real-time CPU, memory, disk, and network metrics to generate predictions. Each Host maintains a local time-series database (`/var/tmp/one_db/host.db`) storing historical data for forecasting. Similarly, every VM has a dedicated database (`/var/tmp/one_db/<VM_ID>.db`) that tracks its individual metrics.
Forecasts rely on OpenNebula’s built-in monitoring system ([The OpenNebula Monitoring System]({{% relref "product/cloud_system_administration/scheduler/configuration/#monitor-alert-configuration" %}})). Periodically, monitoring probes analyze real-time CPU, memory, disk, and network metrics to generate predictions. Each Host maintains a local time-series database (`/var/lib/one-remotes_db/host.db`) storing historical data for forecasting. Similarly, every VM has a dedicated database (`/var/lib/one-remotes_db/<VM_ID>.db`) that tracks its individual metrics.

{{< alert title="Note" type="info" >}}
When a VM migrates, a new database is created on the destination Host, temporarily reducing forecast accuracy until sufficient data accumulates.{{< /alert >}}
Expand Down Expand Up @@ -122,7 +122,7 @@ Forecast database size depends on retention and monitoring frequency:
To optimize storage:

- Adjust retention based on available capacity.
- Monitor `/var/tmp/one_db/` usage.
- Monitor `/var/lib/one-remotes_db/` usage.
- Tailor settings to workload cycles (daily, weekly, monthly).

Changes take effect immediately, without requiring service restarts.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -120,12 +120,12 @@ The monitoring data collected by OpenNebula probes is processed by the monitorin

Each physical Host in an OpenNebula deployment maintains its own dedicated monitoring databases. These databases are updated through the regular Host and VM monitoring cycles:

- **Location**: `/var/tmp/one_db/host.db`
- **Location**: `/var/lib/one-remotes_db/host.db`
- **Purpose**: Stores historical monitoring metrics for the Host

Additionally, for each VM running on a Host, a dedicated database tracks its specific metrics:

- **Location**: `/var/tmp/one_db/<VM_ID>.db` (stored on the Host where the VM is running)
- **Location**: `/var/lib/one-remotes_db/<VM_ID>.db` (stored on the Host where the VM is running)
- **Purpose**: Stores historical monitoring metrics for the specific VM

{{< alert title="Note" type="info" >}}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -258,7 +258,7 @@ This tag can be used at a later time for scheduling purposes, [see more details

### Updating Host Files

When OpenNebula monitors a Host it copies driver files to `/var/tmp/one`. When these files are updated they need to be copied again to the Hosts with the `sync` command. To keep track of the probes version there’s a file in `/var/lib/one/remotes/VERSION`. By default this holds the OpenNebula version (e.g., ‘7.0.0’). This version can be seen in the Hosts by using `onehost show <host>`:
When OpenNebula monitors a Host it copies driver files to `/var/lib/one-remotes`. When these files are updated they need to be copied again to the Hosts with the `sync` command. To keep track of the probes version there’s a file in `/var/lib/one/remotes/VERSION`. By default this holds the OpenNebula version (e.g., ‘7.0.0’). This version can be seen in the Hosts by using `onehost show <host>`:

```shell
onehost show 0
Expand Down Expand Up @@ -321,7 +321,7 @@ Each PCI device reported by the monitor contains the following attributes:
The `IFNAME` attribute is resolved using multiple methods, following this precedence (first match is used):

1. **udev rules** — reads `/etc/udev/rules.d/99-rename.rules`, looking for specially formatted comments. This file is automatically generated by [OneDeploy]({{% relref "../../../solutions/ai_factory_blueprints/deployment/cd_on-premises.md" %}}) and is not intended to be edited manually. It is used to assign persistent names to network interfaces, either for convenience (custom names) or to prevent losing the standard name after VFIO takes control of the device.
2. **File cache** — consults `/var/tmp/one_db/pci_net_names/<PCI_ADDRESS>`, where each file (named as the PCI address) contains the interface name. Created automatically on first resolution.
2. **File cache** — consults `/var/lib/one-remotes_db/pci_net_names/<PCI_ADDRESS>`, where each file (named as the PCI address) contains the interface name. Created automatically on first resolution.
3. **Direct sysfs lookup** — reads the directory `/sys/bus/pci/devices/<PCI_ADDRESS>/net/` to find the kernel interface name.

The cache is cleared when the monitoring daemon restarts.
Expand Down
2 changes: 1 addition & 1 deletion content/product/cluster_configuration/lvm/filemode.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ The LVM Datastore does not need CLVM configured in your cluster. The drivers ref

In case of rebooting the virtualization Host, the volumes need to be activated to have them available for the hypervisor again. There are two possibilities:
* If the [node package]({{% relref "kvm_node_installation#kvm-node" %}}) is installed, they will be automatically activated by the `/etc/cron.d/opennebula-node` cron file.
* Otherwise, manual activation will be required. For each volume device of the Virtual Machines running on the Host before the reboot, run `lvchange -K -ay $DEVICE`. You can also run on the Host the activation script `/var/tmp/one/tm/fs_lvm_ssh/activate`, located in the remote scripts.
* Otherwise, manual activation will be required. For each volume device of the Virtual Machines running on the Host before the reboot, run `lvchange -K -ay $DEVICE`. You can also run on the Host the activation script `/var/lib/one-remotes/tm/fs_lvm_ssh/activate`, located in the remote scripts.

Virtual Machine disks are symbolic links to the block devices. However, additional VM files like checkpoints or deployment files are stored under `/var/lib/one/datastores/<id>`. Be sure that enough local space is present.

Expand Down
2 changes: 1 addition & 1 deletion content/product/cluster_configuration/lvm/lvm.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ The LVM Datastore does not need CLVM configured in your cluster. The drivers ref

In case of rebooting the virtualization Host, the volumes need to be activated to have them available for the hypervisor again. There are two possibilities:
* If the [node package]({{% relref "kvm_node_installation#kvm-node" %}}) is installed, they will be automatically activated by the `/etc/cron.d/opennebula-node` cron file.
* Otherwise, manual activation will be required. For each volume device of the Virtual Machines running on the Host before the reboot, run `lvchange -K -ay $DEVICE`. You can also run on the Host the activation script `/var/tmp/one/tm/lvm/activate`, located in the remote scripts.
* Otherwise, manual activation will be required. For each volume device of the Virtual Machines running on the Host before the reboot, run `lvchange -K -ay $DEVICE`. You can also run on the Host the activation script `/var/lib/one-remotes/tm/lvm/activate`, located in the remote scripts.

Virtual Machine disks are symbolic links to the block devices. However, additional VM files like checkpoints or deployment files are stored under `/var/lib/one/datastores/<id>`. To prevent filling local disks, allocate plenty of space for these files.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -260,7 +260,7 @@ Both Full and Incremental backups are supported by Everpure FlashArray. For Full
Incremental backups are created using the Volume Difference Feature of Everpure FlashArray. This returns a list of block offsets and lengths which have changed since a target snapshot. This list is then used to create a sparse QCOW2 format file which is uploaded to the backup datastore.

{{< alert title="Note" type="success" >}}
You can configure the block size ( Default and minimum 4096 B / 4 KB ) for incremental backups by modifying the file at `/var/tmp/one/etc/tm/san/backup.conf`
You can configure the block size ( Default and minimum 4096 B / 4 KB ) for incremental backups by modifying the file at `/var/lib/one-remotes/etc/tm/san/backup.conf`
{{< /alert >}}

{{< alert title="Warning" type="warning" >}}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -275,7 +275,7 @@ Both Full and Incremental backups are supported by NetApp. For Full Backups, a s
Incremental backups are created by first creating the base full backup from the snapshot however this snapshot is then retained on the NetApp Volume rather than deleted after the backup is taken. When another incremental backup is taken, a new snapshot is taken and both the previous and current snapshots are cloned to new Volumes where they are attached to the host and compared for differences at the block level. These block changes are stored in a sparse QCOW2 file backed by the previous snapshot, which is then uploaded to the backup datastore. The old snapshot is then removed while the new one is retained. When incremental backups are restored, the backing chain is rebuilt before restoring the backup to the VM disk.

{{< alert title="Note" type="info" >}}
You can configure the block size ( Defualt 2097152 B / 2 MB ) for incremental backups by modifing the file at `/var/tmp/one/etc/tm/san/backup.conf`
You can configure the block size ( Defualt 2097152 B / 2 MB ) for incremental backups by modifing the file at `/var/lib/one-remotes/etc/tm/san/backup.conf`
{{< /alert >}}

{{< alert title="Warning" type="warning" >}}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ Default driver actions support the execution of hooks after the main action is s
For example, this is the directory tree of the bridge driver synced to a virtualization node with some custom scripts:

```default
root@ubuntu1804-local-6ee11-2:/var/tmp/one/vnm/bridge# tree ./
root@ubuntu1804-local-6ee11-2:/var/lib/one-remotes/vnm/bridge# tree ./
./
├── clean
├── clean.d
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -354,7 +354,7 @@ The plugins <tm_mad>/monitor_ds + kvm-probes.d/monitor_ds.sh will report the inf

### The monitor_ds script.

The monitor_ds.sh probe from the IM, if the `.monitor` file is present (e.g., `/var/lib/one/datastores/100/.monitor`), will execute its contents in the form `/var/tmp/one/remotes/tm/$(cat .monitor)/monitor_ds /var/lib/one/datastores/100/`. Note that the argument is the datastore path and not the VM or VM disk.
The monitor_ds.sh probe from the IM, if the `.monitor` file is present (e.g., `/var/lib/one/datastores/100/.monitor`), will execute its contents in the form `/var/lib/one-remotes/remotes/tm/$(cat .monitor)/monitor_ds /var/lib/one/datastores/100/`. Note that the argument is the datastore path and not the VM or VM disk.

The script is responsible for getting the information from all disks of all VMs in the datastore in that node.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ Hook Execution Manager configuration is set in `/etc/one/onehem-server.conf`:
|-----------------------|-------------------------------------------------------------------------------------------------------------------------------------------------|
| debug_level | Set the log debug level shown in `/var/log/one/onehem.log` |
| hook_base_path | Base location to look for hook scripts when commands use a relative path (default value `/var/lib/one/remotes/hooks`) |
| remote_hook_base_path | Base location to look for hook scripts when commands use a relative path and `REMOTE="yes"` is specified (default value `'/var/tmp/one/hooks'`) |
| remote_hook_base_path | Base location to look for hook scripts when commands use a relative path and `REMOTE="yes"` is specified (default value `'/var/lib/one-remotes/hooks'`) |
| subscriber_endpoint | To subscribe for OpenNebula events, must match those in `HM_MAD` section of `oned.conf`. |
| replier_endpoint | To send hook execution results (reply to events) to oned, it must match those in `HM_MAD` section of `oned.conf`. |
| concurrency | Number of hooks executed simultaneously. |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,6 @@ For a quick view of any changes in configuration file options in maintenance rel
- `MONITORING_INTERVAL_DATASTORE`: Time in seconds between each Datastore monitoring cycle.
- `MONITORING_INTERVAL_MARKET`: Time in seconds between each Marketplace monitoring cycle.
- `DS_MONITOR_VM_DISK`: Number of `MONITORING_INTERVAL_DATASTORE` intervals to monitor VM disks. `0` to disable. Only applies to `fs` and `fs_lvm` datastores.
- `SCRIPTS_REMOTE_DIR`: Remote path to store the monitoring and VM management script.
- `PORT`: Port where `oned` will listen for XML-RPC calls.
- `LISTEN_ADDRESS`: Host IP to listen for XML-RPC calls (default: all IPs).
- `HOSTNAME`: Hostname to use instead of autodetect it. This hostname is used to connect to Front-end during driver operations.
Expand Down Expand Up @@ -84,8 +83,6 @@ MONITORING_INTERVAL_MARKET = 600

#DS_MONITOR_VM_DISK = 10

SCRIPTS_REMOTE_DIR=/var/tmp/one

PORT = 2633

LISTEN_ADDRESS = "0.0.0.0"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -257,7 +257,7 @@ The error message here (see `ERROR=[MESSAGE="Error monitoring host...`) shows an

```none
Tue Jul 19 17:17:22 2011 [InM][I]: Monitoring host host01 (1)
Tue Jul 19 17:17:22 2011 [InM][I]: Command execution fail: scp -r /var/lib/one/remotes/. host01:/var/tmp/one
Tue Jul 19 17:17:22 2011 [InM][I]: Command execution fail: scp -r /var/lib/one/remotes/. host01:/var/lib/one-remotes
Tue Jul 19 17:17:22 2011 [InM][I]: ssh: Could not resolve hostname host01: nodename nor servname provided, or not known
Tue Jul 19 17:17:22 2011 [InM][I]: lost connection
Tue Jul 19 17:17:22 2011 [InM][I]: ExitCode: 1
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ The internal implementation of Transparent Proxies involves several networking p
* `nft` (`nftables`) to store the service mapping and manage ARP resolutions
* `ip netns` / `nsenter` family of commands to manage and use network namespaces
* `ip link` / `ip address` / `ip route` commands
* `/var/tmp/one/vnm/tproxy` the actual implementation of the “String-Phone” daemon mesh
* `/usr/lib/one/sbin/tproxy` the actual implementation of the “String-Phone” daemon mesh

Below are several example command invocations, to gain familiarity with the environment.

Expand Down Expand Up @@ -165,7 +165,7 @@ default dev br0a scope link
**Listing PIDs of running proxy processes:**

```default
$ /var/tmp/one/vnm/tproxy status
$ /usr/lib/one/sbin/tproxy status
one_tproxy: 16803
one_tproxy_br0: 16809
```
Expand All @@ -182,8 +182,8 @@ There is no PID file management implemented. For simplicity, all proxy processes
**Restarting/reloading config of proxy daemons:**

```default
$ /var/tmp/one/vnm/tproxy restart
$ /var/tmp/one/vnm/tproxy reload
$ /usr/lib/one/sbin/tproxy restart
$ /usr/lib/one/sbin/tproxy reload
```

{{< alert title="Important" type="info" >}}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -247,6 +247,12 @@ Note that the `<hypervisor>` tag should be replaced by the name of the correspon
{{< alert title="Important" type="info" >}}
For KVM hypervisor it’s also necessary to restart the libvirt service{{< /alert >}}

{{< alert title="Important" type="warning" >}}
Do not sync the drivers (`onehost sync`) before the `opennebula-node` packages are upgraded on the Host: the drivers are now copied to `/var/lib/one-remotes` and execute helper commands from `/usr/lib/one/sbin` — both locations are created by the new node packages.{{< /alert >}}

{{< alert title="Note" type="info" >}}
The remote scripts directory is now fixed to `/var/lib/one-remotes` and the `SCRIPTS_REMOTE_DIR` option is gone from oned.conf; a custom value is ignored after the upgrade. The monitoring metrics database is migrated automatically from `/var/tmp/one_db` to `/var/lib/one-remotes_db` when Host monitoring starts. The legacy `/var/tmp/one` directory on the Hosts is left behind: remove it once all Virtual Machines using virtioFS filesystems have been powered off/on or migrated, so their domains reference the new location.{{< /alert >}}

Then update the virtualization, storage, and networking drivers. As the `oneadmin` user, execute:

```bash
Expand Down
Loading