Case #
You face one or more issues while deploying or operating Apache Cloudstack. This article provides high-level guidance on how to troubleshoot Cloudstack on Ubuntu Linux. This article assumes the latest version of Cloudstack, which at time of this writing is 4.16.1.
Solution #
Known issues you may come across with Cloudstack are the following:
- Data Loss on Exported Primary Storage
- Recovering a Lost Virtual Router
- Maintenance mode not working on vCenter
- Unable to deploy VMs from uploaded vSphere template
- Unable to power on virtual machine on VMware
- Load balancer rules fail after changing network offering
- Troubleshooting Internet Traffic
The official Apache Cloudstack troubleshooting guide offers solutions to all above known issues. You should follow this troubleshooting guide for a step-by-step approach. However you should keep in mind the following general troubleshooting hints and tips, when troubleshooting Cloudstack issues:
- Check if your issue is a known issue in the Cloudstack Apache JIRA project or in the official list of issues in the Github project:
- Investigate output from the Cloudstack management server log files. The CloudStack Management Server logs CS web tier (CS portal), application tier and database (MySQL) tier events in /var/log/cloudstack/management/. The CloudStack Agent Server logs its activities in /var/log/cloudstack/agent/.
The following commands can be used to filter these CS logs to only the records referring to a potential issue or error, stemming from either the CS management server or the CS agents (on the respective CS hosts).
grep -i -E 'exception|unable|fail|invalid|leak|warn|error' /var/log/cloudstack/management/management-server.log
grep -i -E 'exception|unable|fail|invalid|leak|warn|error' /var/log/cloudstack/agent/agent.log
grep -i -E 'exception|unable|fail|invalid|leak|warn|error' /var/log/cloudstack/agent/security_group.log
grep -i -E 'exception|unable|fail|invalid|leak|warn|error' /var/log/cloudstack/agent/setup.log
- Investigate a Cloudstack job ID. All jobs running as synchronous or asynchronous transactions in Cloudstsack have a unique job ID. You can find the job ID by doing an initial grep on the CS logs and then make a more granular grep based on the specific job ID(s) which are involved in the error under investigation. An example is shown below for job ID 1532.
grep "job-1532" /var/log/cloudstack/management/management-server.log- Investigate CS database logs which can be queried from the CS portal user interface.
- Activate Cloudstack event log notifications by using the Cloudstack-provided AMQP client. There are two types of events logged in the CloudStack Event Log. Standard events log the success or failure of an event and can be used to identify jobs or processes that have failed. There are also long running job events for asynchronous and synchronous CS jobs. The long running job events log when a job is scheduled, when it starts, and when it completes. To configure CS AMQP for event log notifications to be sent to external apps and services, consult the following URL: https://docs.cloudstack.apache.org/en/latest/adminguide/events.html#amqp-configuration.
- Optionally, you can also enable the CS Apache Kafka framework. To configure CS Kafka for event log notifications to be sent to external apps and services, consult the following URL: https://docs.cloudstack.apache.org/en/latest/adminguide/events.html#kafka-configuration.
- If all else fails, try to send your Cloudstack error/issue to the official Cloudstack mailing lists and IRC channels:
Reference on how to troubleshoot Cloudstack on Ubuntu Linux #
Official Apache Cloudstack troubleshooting guide: https://docs.cloudstack.apache.org/en/latest/adminguide/troubleshooting.html
