VMware vCenter Log Insight Series Part3 - Collecting Logs

The most important step in analyzing logs is...collecting logs. In this post I will briefly guide you in how you can collect logs in vCenter Log Insight.
As previously said, Log Insight features a syslog collector, this means that it can ingests logs from literally anything: ESXi hosts, physical infrastructure, virtual machine guest O.S., application logs, etc.

While configuring ESXi hosts and vCenter log collection is an automated task that can be easily performed from within Log Insight itself as seen in previous post, other devices must be manually configured in order to send logs over to Log Insight.

Here's how to collect logs from other sources rather than ESXi hosts and vCenter Server:

1)Windows operating system:

vCenter Log Insight provides a native agent to be used on Windows to collect logs. Windows agent can be retrieved from Log Insight Administration -> Agents page. Agent is a small executable that can be installed on any Windows machine you want to retrieve logs from.

After agent installation has succeeded logs are immediately sent to Log Insight to be collected. Please check Administration -> Agents page to verify that your Windows server has been successfully registered.

Agent is installed by default in C:\ProgramData\VMware\Log Insight Agent. liagent.conf is the configuration file in which you can configure several parameters related on logs collecting such as what kind of logs to collect and what event severity to collect.
Windows agent can also be used to collect specific applications logs from software, residing on the same machine on which agent has been installed, such as Microsoft SQL logs, Apache logs, etc.
This can be done specifying in liagent.conf file the path where to retrieve application specific logs. For further informations regarding app logs collecting have a look at this Log Insight documentation page.

Agent configuration can also be performed in a centralized way via Log Insight Administration -> Agents page. These settings are server-side and will be applied to every client connected to Log Insight, these settings are more authoritative compared to the ones set on client-side, this means that if same setting exists both on client-side and on server-side, the latter is the one that is valid. Otherwise client and server-side settings are merged to create a resulting configuration.

2)Linux operating system:

We will use Rsyslog to send logs from Linux over to Log Insight. Rsyslog can also be used to create a centralized logging server when deploying Log Insight in an hub-spoke architecture.
Rsyslog installation and configuration is quite simple. On RHEL 6 and later is installed by default, on previous versions and on RHEL flavoured distros you can install it using:

 yum install rsyslog  

On Debian and similar distros, such as Ubuntu, installation can be performed with:

 apt-get install rsyslog  

Once installed configure it by editing /etc/rsyslog.conf file. I prefer to use UDP for log transferring even if TCP is also a viable protocol. To enable UDP modules on Rsyslog edit rsyslog.conf by removing comments ("#" at line start) on:

 $ModLoad imudp  
 $UDPServerRun 514  

Add this line at the end of file in order to send logs over to Log Insight:

 *.*  @Log_Insight_IP_Address_or_FQDN:514  

Save file and restart Rsyslog service. If needed add an exception to Linux firewall in order to open port 514 UDP.

 iptables -A INPUT -m state --state NEW -m udp -p udp --dport 514 -j ACCEPT  

Verify that Rsyslog will be automatically started on boot. On RHEL-like distros you can do this with:

 chkconfig rsyslog on  

3)Physical Devices:

Physical devices such as switches, routers, storage nodes and servers typically support sending logs to a remote destination. This of course vary depending on device model and manufacturer and the appropriate documentation must be consulted in order to find out how to offload logs to an external syslog.
This process is usually quite simple and can be done either via a graphical management console either via CLI.

As a general example here's how to configure an HP 29xx physical switch to send logs over to Log Insight.

Connect to switch management interface either via serial cable either via SSH. Enable offsite logging with the following command:

 logging <Log_Insight_IP_Address_or_FQDN>  

That's all. In next post we will get some useful informations by analyzing logs.

Other articles in this series:
VMware vCenter Log Insight Series Part1 - Introduction
VMware vCenter Log Insight Series Part2 - Installation and Configuration
VMware vCenter Log Insight Series Part3 - Collecting Logs
VMware vCenter Log Insight Series Part4 - Creating Analytics