Randy Lastinger – RWSP 00018
Peak Security
Real World Security: Attack, Defend, Repel
7 September 2010
Syslog:
A Defensive Security and Stability Mechanism
The syslog protocol is defined as “A protocol for transmitting event messages and alerts across an IP network” (PCMag.com). Wikipedia defines syslog as “a standard for forwarding log messages in an Internet Protocol (IP) computer network. It allows separation of the software that generates log messages from the system that stores the messages” (Wikipedia). Eric Allman developed Syslog in the 1980s to be used for the Sendmail project, but was later adopted as an industry standard by Unix and Unix-like systems. There are now several iterations of syslog within the industry on other operating systems, software packages, and commonly found in network appliances such as routers. In 2001, syslog became an IETF standard protocol where it is documented in RFC 5424 (Wikipedia).
Syslog is deployed in organizations and governments across the world however, many, if not most, implementations of syslog appliances or software has been strategized to only fulfill certain requirements, and in turn, is not utilized to its full potential which does not allow the organization to receive full Return on Investment or ROI. Often times, syslog is deployed specifically for the reason of meeting regulatory compliance regulations such as Sarbanes-Oxley, also known as SOX, PCI DSS (Payment Card Industry Data Security Standards), EI3PA (Experian Independent Third Party Assessment), and HIPAA (Health Insurance Portability and Accountability Act), and even State or Federal regulations depending on the business requirement. Syslog is also used to simply troubleshoot network issues. A good syslog implementation has the potential to drastically increase an organization’s security defense posture and stability if fully implemented while continuing to meet or exceed these regulatory requirements. Because security and IT professionals tend to look at syslog in a blinded manner, it tends to be an unknown gem in the security and IT industries.
Problems with Syslog
Syslog implementations will quite frequently become troublesome and difficult because operating systems, network devices, and applications have different logging formats and heading identifiers. Currently, it seems that an organization needs to hire someone to solely implement and manage syslog. The skillset of this position will need to be someone that is familiar with the logging formats of the different devices in the environment, has a strong scripting background, knowledgeable of regular expressions, an understanding of logs, and knows the syslog product inside and out however, this person typically does not necessarily know all uses and potential capabilities of syslog. Because of this, syslog seems to be a scary and expensive beast to businesses and administrators for this reason.
Syslog vendors unfortunately use the fear and complexity of syslog to their advantage, and have a tendency to develop the syslog products so that it almost seems to take a doctorate in nuclear engineering to figure out how to deploy and manage these devices or applications. Businesses tend to sell the packages to customers with a caveat that the product is only fully supported if the vendor or vendor partner deploys the product on behalf of the customer which drives initial implementation and support costs through the roof. They do this by developing their own logging headers, scripting languages, and regular expression formats that do not meet any of the standards tending to make syslog seem more terrifying than it should be. Even though the vendors should know the product inside and out and should be able to make recommendations for a full deployment to include security defense measures, these companies often tend to deploy syslog at a regulatory compliance standard and leave it be. The employee that is trained on syslog is typically minimally trained to maintain what has been deployed and will not try to further the deployment to obtain better ROI on the product.
Many organizations will pay a third party vendor to manage syslog for them. In these instances, the company pays the vendor a fortune to have a syslog device placed inside the customer’s environment, and the customer is expected to configure whatever devices is needed to be logged according to the customer’s understanding to the syslog device. The vendor then has someone at its security operations center to monitor the logs and send notifications on potential events. This solution is also very expensive, but requires little knowledge of syslog. The problem with this solution, besides cost, is trust. Who is looking at the logs? What else is the vendor’s employee doing with those logs? Has the vendor’s employee been trained well enough to know what he/she is looking at? Despite the answers to these questions, it does seem to provide the customer a piece of mind that someone is monitoring it those logs, and someone within the company is being paid to manage the vendor. It also provides piece of mind in that auditors tend to look be okay with this solution for meeting regulatory compliance obligations.
Another problem with syslog is storage space. Many organizations, especially organizations where administrators do not fully understand the benefits of syslog, will typically use the reasons of storage costs to fight the implementation by providing the cost of the most expensive disk possible. This is not a false statement. Storage space for these applications can be expensive, but to help reduce cost, a well-defined syslog application does not need extremely fast enterprise level disk space although sometimes, this is the only disk available. In fact, as important as a good syslog program is, syslog application data could typically reside on the slower near-line or tier 2 or 3 storage if available, because it does not need to be replicated, nor does it need to be a tier 0 application in the event that the disaster recovery plan needs to be implemented however, syslog should be implemented and is very important to the vitality, security, and stability of the organization in both production and disaster recovery.
In order to reduce the cost of disk space, the administrators and security professionals should consider the size of the environment being logged, what devices are in scope, and what exactly should be logged in order to be able to make some sort of calculation of the potential rate of change. Some ideas of what should be logged will be mentioned later in this paper. Once determined what this disk space should be, an archival and backup maintenance plan should be created. Many regulatory compliance regulations such as PCI DSS, the payment card industry standard, and EI3PA, the Experian security standard, require a log retention policy of at least one year. This does not mean that at least one year of data has to be online at all times, but does mean that the log data should at least be able to be recovered from tape from at least one year ago. This means that a backup strategy now has to be organized and tested which incurs a little more expense, but is typically minimal and adding this application data to the current backup plan should require minimal effort. Implementing the archival plan is what will typically take some time and effort as this strategy should include backup data to tape, archiving data to ensure that only a few weeks or months is available online depending on the organization’s policy, and maintaining the data log files as many syslog applications will log the data in a backend database which will build a large transaction log file. Once the database is backed up, remember that the data that was archived and removed from the data may leave white space which will also need to be shrunk and removed in order to maintain storage.
Misconceptions of Syslog
IT and security professionals often have a tendency to view the importance and need of syslog in a negative manner. Often times this is encouraged by management thinking of it as an unnecessary expense and waste of time. Because of this, many common misconceptions about syslog travel through the industry which makes syslog a difficult sale.
One of the most common misconceptions of syslog is that the best or only real use for syslog is for network debugging and centralization of network device logs. Every organization should have at least one network switch, firewall, and router that outputs thousands of logs every hour of every day. It can become very expensive to hire someone to sit there and look at the logs for these devices day in and day out, so syslog comes in very handy with these devices because it provides a centralized look at all logs from all devices for troubleshooting network connectivity issues, ACL violations, proper and improper tear downs of packets, etc. however, if the logs of these devices are not configured properly for logging, you will also catch many logs that are not needed. A proper implementation of syslog cannot only assist network administrators and engineers in troubleshooting these issues, but in some cases can also help determine potential attacks to the environment which will be talked about later in this paper.
Another common misconception about syslog is that syslog is used for spying on network and system administrators. This misconception does have some truth to it in that if properly configured it could actually be used to find out what system and network administrators are doing, but is usually not truly the reason that syslog is being implemented. Because of this misconception, security professionals do receive a lot of fight and push back from systems and network administrators when implementing syslog and often finds the implementation run slow or late due to lack of cooperation with network and systems administrators which is why it takes a good, understanding security professional to get buy in from the administrators so that they will be willing to work with you. This is often an easy task for a security professional with a proven strong technical background to defend by providing other useful syslog techniques to the systems and network administrator that will be discussed later.
A security professional may hear many excuses from systems and network administrators that he/she must be ready to answer for within an environment. For example, typical Windows administrators will suggest that syslog is not needed because Microsoft Operations Manager (MOM) is used to centralize the windows event logs, or an additional syslog resource will put too much overhead or stress on the network or system resources and could cause of denial of service. If the security professional has analyzed the environment properly, the implementation of the syslog should be fairly easily defended when these allegations are made.
Another common misconception is that syslog is only useful for meeting regulatory compliance obligations, and if it were not for regulatory compliance we would not be spending money on it. People that have this misconception often think that syslog is a waste of money as well because many security professionals either do not think about the full potential of syslog and capabilities, or does understand the full potential power of what it can do, but not sure how to implement it. Some of these strategies might be discussed later, but because most syslog products have their own individual implementation standard, an example may not be available.
Commonly Known Uses of Syslog
One of the many commonly known uses actually is network debugging which is also mentioned as a common misconception. To clear things up, the misconception of network debugging is that many people think that syslog is only useful for network debugging. In fact, syslog can be much more useful for network administrators than network debugging, especially if you have a team of network engineers. Configuring and filtering proper logging on the network devices themselves can help identify issues such as interface statistics, port flapping, and even configuration changes. There are other uses of syslog that are less common with deploying syslog across network devices that will be mentioned later.
Debugging systems and identifying system errors is another commonly known use for syslog. Contrary to popular belief, a properly configured syslog device can tell a lot about a system. It can provide full detail of boot statistics, bad sectors in memory or on a hard drive, operating system issues, etc. Syslogs can also sometimes catch the last log that occurred right before the dreaded “blue screen of death” that Microsoft systems are famous for. Knowing this information can give administrators enough forethought to determine when a system might have a major issue before the issue permanently occurs. A really good syslog product and implementation will even go as far as alerting the team responsible for those systems by email or text message to let them know that there is a potential issue. This forethought can be very beneficial in maintaining a company’s stability and productivity and maintaining growth in the bottom line, but must begin with having a proper configuration of logs on the server devices themselves. It is not uncommon for administrators to limit the logs or longevity of how long logs remain on the system, so it is not uncommon for these logs to be missed which in turn prohibits a good syslog security program from being successful. Another useful tip on syslog is that even though the logs may have been rotated out of the system itself, the log is still available on syslog
Security professionals use syslog the most as a security management device. This provides a central area for security professionals to track not only errors and events within the network and systems, but also ensure that the minimal regulatory compliance logging is in place. Most regulatory compliance standards for syslog are very similar, so often times a security professional will pick the compliance regulation with the most controls and meet those obligations as minimal. This typically will meet the other requirements as well. Auditors will tend to let some things slide if not all syslog regulatory compliance controls are not met as long as at least one of the regulatory compliance standards are met. The main syslog item usually requested by auditors is an audit log of changes or commands run on the specific systems that are in scope for those compliance regulations. The systems that are in scope usually include authentication systems, application servers, the devices where the data rests, and the network devices.
Syslogs are commonly used to gain information for legal issues as well. Although legal issues are typically not thought of during syslog deployments, it is true that they can be used for this purpose. This is why most security administrators insist that logs become part of the backup cycle, not to mention that some regulatory compliance standards require a log retention policy of at least one year. Many people ask how syslogs can be used as legal evidence in a case. It has been discussed in this paper already that one of the common uses of syslog is the ability to log changes or commands run on a system. These logs also include a time/date stamp and username of the employee that made the change or ran the command. So if you remember the common misconception that a syslog is only used for spying on the administrators, there is some truth to this however, spying is not the only purpose for syslog. In the security industry, this is more known as accountability instead of spying when used for this purpose.
Uncommon Uses of Syslog
Syslog has many other uses that are commonly not thought of by security professionals. If these uses are thought of, it does not seem to be typically deployed within the syslog environment. These uses can be used to help strengthen the security defense posture of the environment, help determine and maintain stability of the environment, help speed up if not automate some investigative processes, and work in conjunction with other IT and security mechanisms deployed to protect the security, stability, and productivity of the environment.
IDS/IPS Correlation
Many environments have intrusion detection/prevention systems, also called IDS or IPS, deployed within the organization. One of the most common IDS systems is a product by SourceFire called Snort. Snort is a powerful package that has the ability to detect “probes or attacks, including, but not limited to, operating system fingerprinting attempts, common gateway interface, buffer overflows, server message block probes, and stealth port scans” (Wikipedia).
Many security professionals believe that having an intrusion detection/prevention system on the network is the only solution needed to detect anomalous activity and is the only/best means to determine threats on a network level. The truth is that although intrusion detection/prevention systems are important within a company’s security posture, the system is only as good as the administrator. Intrusion detection systems can log many false positives and therefore all findings within an intrusion detection system need to be reported to an incident response program to be investigated. This investigation process can sometimes be accelerated by a properly deployed syslog program because if proper logging is enabled, the logs could possibly show an unexpected flux in activity if being attacked or a change to the system or systems being attacked that could be considered abnormal. If the company is using a product like OSSIM, Prelude, or other SIEM products, the administrator may be able to perform some scripting that would cross-correlate events between the IDS and syslog devices and open a ticket automatically while alerting administrators of a potential finding.
Intrusion detection systems also have a problem of false negatives meaning that the system may not see an intrusion that actually exists because of improper configuration, or the traffic actually looks to be valid. In some cases, an attacker might maybe fragmenting packets and the system is not able to recognize the traffic as well. A properly configured syslog should be able to see any abnormal activity via the log files of the attacked device in the event that an attack is occurring. This provides a second measure of security in which the syslog has now turned into a host intrusion detection system and have doubled the return on investment. Syslog cannot replace a product such as Snort or other intrusion detection systems, but can definitely help assist in verifying accuracy and potentially see possible attacks that intrusion detection systems may miss.
Monitoring Correlation
All organizations are concerned about the stability of the systems and product uptime, especially those organizations whose revenue is generated via the internet. Many applications exist today to assist these organizations with this important task. Some methods of monitoring are via a protocol known as SNMP or Simple Network Management Protocol, such as Cacti, and some methods use client/server, such as Nagios. Both methods are good depending on the environment, but both also tend to have problems in that they can sometimes report false positive or false negative information depending on how the administrators configured the monitoring applications. Administrators can commonly see that when the client is not reporting to the server for whatever reason, the server will begin to throw messages stating that the server or service is down which usually forces administrators into a panic to find out what happened only to find out that the monitoring application temporarily lost connectivity to the server. Sometimes the monitoring server or service will go down or the email mechanism may cease to function for whatever reason and the alerts are not able to get to the administrators. With a proper syslog deployment, basic important services and other functions of a server can also be monitored via syslog as a secondary measure by configuring a simple regular expression looking for the specific term that the particular operating system uses. Typically, monitoring for the word “service” will usually work on a windows system, and with Unix systems words like “daemon” or “service” will sometimes work as well. Syslog does not have the capability of replacing a monitoring system, but can definitely assist in some basic monitoring activities.
Syslog also has the ability to sometimes take monitoring to another level in that it can sometimes report why the monitor was generated. Most administrators are asked why a server or service failed by managers when a monitoring alert is generated. In order to find these answers, the administrator could be digging in logs for quite a while until being able to generate an answer. A proper syslog program that monitors for specific error codes or key words will often be able to alert the administrator of the log that generated the alert, and sometimes the logs that were generated leading up to the alerts. This will help administrators identify problems faster and sometimes be proactive and resolve issues before they become a major problem.
Monitoring can also be used to assist syslog in that attackers will often turn off logging once the system is compromised. If monitoring is also monitoring logging services, the attacker would need to turn off both logging and monitoring in order to successfully remain undetected. A good syslog program will also be able to detect when itself is turned off in that many, not all, syslog packages have the ability to script and detect when logs have not been sent to the server when logs are expected. Some syslog packages have this function built-in. The syslog server will then generate alerts that states that logs have not been received from a particular device in however many minutes.
System Authentication Tracking
Authentication issues such as a user account lockout are not normally thought of as security anomalies, but they could be. Security professionals should know the most common attacks originate from within the organization, and therefore like to track user account authentications and lockouts in order to determine unusual trends or if a user is attempting to access a device that is unauthorized. Most syslog applications have the functionality to track this functionality with a little scripting ingenuity.
One major vendor’s engineering team was able to use their software to script a method within their logging software that had the ability to detect when a user account was locked out based on the company’s administrator providing accurate information from the Group Policy settings. When implemented into the customer’s environment on specific servers, this script was able to determine the misuse of certain high level accounts, and several services or scripts that ceased to function when user accounts were locked out. This was able to provide the customer much more information about how accounts were being used and provide better stability and tracking of their devices.
Application and Web Application Monitoring
Most of this paper has discussed how syslog can be used to monitor system logs, detect host based anomalies, debugging, etc. however, syslog is often overlooked as a means of monitoring web application and application logs whether the code be from a third party vendor or home grown code. Third party vendors software products, especially products that are used for production or audited environments such as accounting systems, provides the customer the ability to set specific logging levels in order to ensure that all regulatory compliance regulations are met in the event that these regulations exist for the organization. Within these logs, the application could show that an administrative username created a new account, reset a password, changed specific data sets, or output a report to a file. This type of data is interesting not only to a security professional, but also for legal review, and organizational stability and reporting. If logging is set high enough, the application or web application will log the exact commands that were submitted to the applications.
When logging home grown applications, it is important to have a standard logging level set within the SDLC, or System Development Life Cycle, that is agreed to by the Development team, Operations team, and Security team so that everyone has an understanding of the logging levels of all applications in order to determine best logging practices for that particular group and organizational needs. In order for syslog to be effective, the security team should attempt to get as much verbose logging in the applications as possible in order to be able to use application log monitoring as a potential defense mechanism and stability monitoring for the security syslog program. It is important to remember that a good application syslog monitoring program includes logging both error logs and non-error logs because both logs can provide useful information.
Assuming that the SDLC provides verbose enough application logging, these applications, both home grown and third party, should be able to help determine and detect errors and bugs within the software and notify the proper administrators when these events occur. The security team will need to work with both teams in order to ensure that the syslog application is capturing the proper key words within the application log events in order to be successful. This can be done by use of regular expressions. This will help the development and systems teams track issues and become able to properly respond to any problems within a timely manner.
A good application logging and syslog program might also be able to identify specific attacks against the application regardless of whether the attacks are submitted via POST or GET methods. These attacks could include brute force attempts, cross site scripting or XSS, sql injections, CRLF injections, etc. Additionally, these logs can sometimes be found in non-error logs as the software may find these attacks to be valid inputs if the application and variables are not programmed or sanitized properly. These attacks can be logged by the application logging the input parameter or URL parameters passed into the application. For example, an attacker might attempt to perform a basic SQL Injection attack against a web application by attempting to pass a simple “’ or 1=1- -“ to the application. The log should show that “’ or 1=1 - -“ was passed to one of the form fields in the application and provide the IP address from where the attack originated. If the syslog program is monitoring these logs with a regular expression looking for this input value or other attack type input values, this attack might be capture. Capturing this information is important because it now alerts security professionals to know that an attack attempt was made on the application and a ticket should be escalated for investigation. Additionally, the development, systems, and security team will know what to investigate in order to more easily determine if the attack is successful and the amount of damage of the attack in the event of some success in order to follow proper incident response procedures.
Bot attempts can also be detected by effective application logging programs. Wikipedia defines bots as “software applications that run automated tasks over the Internet” (Wikipedia). These bots will scour the web looking for specific software issues, but are very repetitive in nature and easy to detect with a good logging program. This is easy to detect because even though syslog alerts will be set off, proper identification of logging will allow administrators to more easily determine the cause and determine that it is a bot during the investigative process. The administrators will also know what was scanned and will be able to run tests against the system to continue to determine that the application is not vulnerable to application vulnerabilities that could lead to another issue later.
Database Logging
Typically, most database logging is basic. Most syslogs are configured to monitor only the commands that are run by specific users from within the console and who logs into the database because regulatory compliance regulations such as SOX require it. Other problems within the system can occur as well that will often get missed because of improper logging and syslog configurations.
While it is important to log manual changes that are executed on the system, and required by many regulatory compliance standards, it is also important to log queries run by service accounts or accounts used by applications. Developers and administrators understand that most service accounts and application login accounts are used for automated procedures that are supposedly pre-determined by scripts or applications, and it is often assumed that for this reason, the automated processes run by these accounts are generally safe and do not need to be logged. Attackers know and realize this also, and therefore assume that this will be an attack vector that could be used with little detection.
It was mentioned earlier that with proper application logging, certain attack vectors like SQL Injections could possibly be detected by proper configurations of logging and proper monitoring of syslog. The SQL injection statements such as “’ or 1=1 - -“ is one of those statements that may also show up in the database logs under an application account that is generally thought of as safe and ignored in syslogs. These types of attacks could appear in both the application and database logs because the information input into the form input or URL strings are typically passed to the database from the application. If an attacker is able to determine that an application backed by Microsoft SQL is vulnerable to SQL Injection, the attacker may decide to further his attack with a known tool called SQL Ninja. One specific feature that SQL Ninja offers is the ability to spawn a reverse shell of the system where the database application server resides. Microsoft SQL Server has an extended stored procedure called “xp_cmdshell” which provides command-line access to the operating system. In fact, most database application servers have some sort of procedure that will provide this functionality. Microsoft typically recommends that the “xp_cmdshell” extended stored procedure be disabled and/or deleted when securing SQL Server. Even though SQL Servers are not often secured properly, SQL Ninja has the ability to re-create the “xp_cmdshell” extended stored procedure on the database and would be logged in the database logs under one of these automated accounts. Proper logging of these accounts would also help administrators to know when these events occur considering that the server was properly secured from the beginning. Even if the server was not properly secured however, a well configured syslog monitoring database logs should still be able to alert when this extended stored procedure has been called and know whether or not it was called manually or via one of the supposedly safe automated application accounts or service accounts.
Monitoring Non-Error Logs
Some of the most commonly forgotten logs when implementing syslog programs are logs that do not contain errors. This has been discussed once already when discussing the benefits of syslog with applications. Non-error based system logs are also often forgotten. When talking about non-error based system logs, these would be log files that include apache access logs for example or logging when a user successfully logs into a system, IIS access logs, etc. These logs also contain important information that could assist in implementing a successful syslog program and assist in increasing an organization’s security defense posture.
Monitoring web server logs can also provide a secondary means of information for detecting some of these attack vectors as well. Web server logs like apache and IIS will often log very valuable information with default configuration. For example, an apache log file will provide information such as the IP address that accessed the server, date and time stamp, the URI accessed, and a web server error code such as 200, 302, 404, or 500. Normally, 200 and 302 error codes can be found in the Apache access logs, and 404 and 500 would be logs found in the Apache error logs. Knowing that this information exists, what the logs mean, and how to use these logs to your advantage could be valuable in implementing a successful syslog program and in expediting the turnaround time of an investigation of a detected attack.
Often times web server administrators will unknowingly misconfigure a web server and accidentally have a web application that is meant to be used internally only to the public internet. Monitoring non-error syslogs like Apache access logs can be very beneficial in this case as the administrator has no knowledge that the application is accessible externally until someone alerts him or her, or syslog is able to tell him that someone outside of the company accessed the site. For example, a company maintains their press releases in a database and writes an application meant “for internal use only” to help the marketing team manage them. If the web server administrator misconfigures this application, anyone on the internet could come across this page and hit a button that says “Delete All Press Releases.” Needless to say, when the company realizes that the press releases are gone, and investigation immediately begins. The company could spend days trying to figure out how the press releases were deleted. If the syslog is properly monitoring the web access and application access logs, the administrator could be alerted by writing a script extracting the IP address, comparing to the internal IP scope, and passing anomalies to the syslog. The administrator would then know that someone outside of the network, or maybe even inside of the network, browsed to the site and pressed the button. More than likely, spidering and bot programs would have found the site before the button was pressed which should have alerted the administrator of the misconfiguration first.
Many companies use applications like phpMyAdmin which is a web based application used to assist in administering and maintaining MySQL database applications. It has already been mentioned that misconfiguration of these applications can lead to powerful applications being leaked to the internet by accident creating a major exposure to the organization’s data and applications. Most of the time, these applications can be searched for and found through a carefully crafted Google search such as "Welcome to phpMyadmin" "root@localhost", and the phyMyadmin can then be viewed by an attacker or random innocent bystander. More often than not, a bot will be making it’s way through the internet logging where these applications are found. These bots will create logs in web server which will most of the time show up in the web server error logs however, in the event that the web server administrator made a mistake with an application like phpMyadmin, the logs will show up in the access log instead because the site actually does exist and was misconfigured by the administrator. If the access log is not logged, the administrator and organization will have no idea that this application has been compromised and can put the organization at great risk. A good syslog program will use a similar technique as mentioned before and can help find these issues and send alerts to the administrator to help defend against these threats. These attacks will more than likely be logged in incident response investigations, but having this implemented successfully in the syslog program is definitely beneficial to an organization’s security defense posture.
Network Device Brute Force
Network devices such as routers, switches, and firewalls are very important to the security of any organization’s success. Because of the importance of these devices, it is important that these devices are secured, monitored, and logged properly. Most of the time, these devices are found to be logged, and the logs will be archived daily, but never viewed unless the organization was having network issues that day. A proper syslog program can actually provide a little more information about an organization than managers, network administrators, and security professionals are aware of.
As most IT and security professionals know, one of the most common attacks against network devices is brute force attempts. This technique holds especially true against network devices that are on the perimeter of the organization. Brute force attacks are typically used by attackers when an attacker cannot find a vulnerability to exploit. The attacker will then queue up brute force password attacks or dictionary password attacks against the device. It is not uncommon for these types of attacks to not be logged properly or overlooked if they are logged. Sometimes the problem is that the device is not configured properly to log these types of events to the syslog device. If the organization is using a Cisco router on the perimeter, an organization may consider using a configuration similar to “login block-for 60 attempts 3 within 60; login on-failure log; login on-success log.” This configuration on the device would allow all logins, success or failure, to be logged to the syslog however, not all of these attempts should be considered events to generate alerts. The events that would be important are those failed or successful login attempts that are attempted by devices outside of the ACL scope which could be monitored by a script comparing the IPs in the ACL scope with the internal IP scope.
There is a major part of this configuration that needs to be discussed. The part of the configuration that states “login block-for 60 attempts 3 within 60” is the interesting event in this configuration because it should not only generate a syslog event, but also helps to increase the security of the device by helping to deter or log false positives and false negatives to any bots or automated brute force password or dictionary password attacks. This part of the configuration says that if a failed login occurs three times within sixty seconds, then place a block on the account for sixty seconds before it is allowed to login again. This causes possible false positives and false negatives to brute force password and dictionary password attacks because it is possible that the correct password may have been attempted while the lockout policy was in effect. Violations of this configuration should definitely be logged to syslog and generate alerts because this is typically a sign of an attack and merits investigation to determine if the attacking IP should be blocked or whatever action is determined to be taken. Knowing this information and being alerted on it can definitely increase the security defense posture of the organization.
Other events that should be logged are commands and access control list, or ACL, violations. These logs are typically thought of because of regulatory compliance requirements, but are often ignored. Many products do not have the ability to be able to alert and correlate if a user submits a command that should not be able to, or escalates privileges even though the user should not know the password, but those syslog applications that allow advanced scripting or regular expressions do have this ability. ACL logs are commonly known to be violated and logged, but are often ignored and not viewed. These logs should also be alerted depending on the source of the violation and the frequency of the violation. This can provide valuable information as well and often let an administrator know of any potential internal attacks, or if a device needs access to something that was unknown. These alerts should be discussed between the security team and network team if this option is available within the product.
Device and System Log Tampering
As most security professionals know, device and system log tampering is usually evidence of malicious activity however, not all tampering is accidental. Occasionally log tamper is performed accidentally or even as part of a maintenance plan, but is more often evidence of malicious activity than the latter. Once a breach occurs, attackers are notorious for covering their tracks by deleting the entire log that shows evidence of the attack, or removing only the entries in the log files that show the attack and attack success so that administrators will not have any idea that the device or system has been breached. It is also not unusual for attackers to disable logging once the system has been compromised as well. Syslog programs can help detect these events as well and will also help administrators and security professionals respond to this breach faster.
Many syslog packages utilize the client/server architecture, such as OSSEC, in order to get the logs. This technique can be helpful in the event of log tampering for two reasons: 1) syslog application servers typically tends to log and alert the administrators when it has lost connectivity to one of the clients which is beneficial because the administrator can immediately investigate why the client machine is down, especially if the monitoring package says that the server is up; and 2) if an attacker turns off the system log first before the syslog application client, the client will be able to send the notification that the local syslog has been disabled. Once the syslog sends alerts that these functions are disabled, these events would immediately be elevated to the incident response programs for investigation and be able to take immediate action.
Other syslog packages require sending all of the logs over the network to the server, such as Cisco MARS, where the filtering and alerting takes place. This methodology can also detect device and system log tampering although, maybe not as efficient at doing so as a client/server architecture. Some syslog applications using this method have built-in internal intelligence to determine when devices have not sent logs to the server when the server is expecting them and alerts can be sent based on those events. Other syslog applications that utilize this method may not have the intelligence to detect when systems have not been reporting for a while, so a script would need to be written in order to make this determination. Packages that may not have this intelligence would be products like the Syslog service that usually comes with the Unix operating systems. A script would need to be written that looks through the log and is able to real-time calculate the last time that the syslog has heard from the expected system or device. In the event that the system or device has not communicated within the timeframe that the script allows, an alert would need to be generated by the script so that the administrators can investigate. This possibility is solid as well. The disadvantage to this method than the client/server method is that if the attacker disables the syslog service on the device or network, it is not guaranteed that the disabling of the service would be logged to the syslog application server. The organizations would solely be relying on the monitoring to system to detect that the syslog service has been disabled until the syslog application server is able to realize that no logs have been received.
File Integrity Checking
Many syslog applications are now including the capability of file integrity checking. File integrity can be defined as “The principle that completeness, original file order, and unbroken custody of the records in a filing system must be maintained for a record series to maintain legal and intellectual integrity” (osulibrary.oregonstate.org). The goal of file integrity checking is to detect when certain files have been changed due to an attack, by an administrator, or by an application. This can also include registry changes on a Microsoft Windows system (ossec.org).
Finding a syslog application server that has the ability to perform file integrity checking has recently become more important because of its adaptation into regulatory compliance standards such as PCI DSS, EI3PA, and HIPAA. It is also important to the organization as it has the ability to detect and alert personnel when important organizational files have been changed. Some applications may have the ability to tell what was changed within the file as well. This would be very important to an organization in the event that the data that was changed reports malicious or false information which has been known to occur within accounting files, medical records, credit files, etc., and would be able to increase the investigation time as the software would specifically tell the investigators that employee A changed file A at this specific time and date. Most application syslog programs with this capability do not show who changed a file, but might alert that the file was changed and when it was changed which is a good start of the personnel responsible for the investigation of this information as this information can also be cross correlated with the login events of the system to determine who was logged in to that device at the time the file was changed.
Another major benefit of file integrity checking goes back to network and system log file tampering. If the syslog application is configured to monitoring file integrity within these files, the application might be able to detect unexpected changes to the files as well. This type of monitoring can cause many false positives however because it is in fact a log file which means that file is changing constantly. In order for this type of monitoring to be successful for file integrity checking, the security professional or administrator would need to set specific guidelines as to which the file is being checked assuming that the syslog application is robust enough to support this, or the administrator’s scripting knowledge is strong enough to support this. Not all syslog applications that have file integrity checking capabilities may be able to support this functionality. If the syslog application does support this, the administrator would need to set a guideline that monitors two things: 1) Monitor for a decrease in the log size. Most logs will not decrease in size until the system performs log routine log maintenance as defined by the system or applications administrator. Knowing this maintenance schedule will help determine if the logs decreased in size at an unexpected time, or if the decrease was performed by the system itself. If the logs decreased in size unexpectedly, it is possible that the logs were decreased because of an attack or an accidental keystroke by an administrator; and 2) Monitor for any significant increase in size. Buffer overflows, brute force attacks, and heap overflows can sometimes cause the logs to grow at an unexpected exponential rate. Being able to detect that the logs are growing too fast could cause an alert and trigger the administrators to investigate the activity immediately. These methods may require scripting if the application can support this capability, but can definitely bring valuable information when defending against attacks against the organization.
Many people will ask “What if my syslog application package does not have file integrity capabilities? What do I do then?” Rest assured there are other software packages available that have file integrity checking capabilities in the event that syslog application packages do not. Some of the file integrity applications were probably available before syslog applications were providing capabilities of monitoring file integrity. Of these applications, three of the more common are: 1) TripWire – TripWire is a commercial application that has the ability to not only monitor file integrity of the configured files, but will also let the organization know what file was changed, who changed, when it was changed, and what has changed within the file. TripWire is compatible with Unix and Windows systems; 2) Afick – Afick is an open source application that simply performs file integrity checking on configured files and alerts when the files were changed. Afick is compatible with both Windows and Unix systems; and 3) Osiris – Osiris is also an open source application with file integrity check capabilities, but if configured properly can also look for trojan horses and rootkits and can send alerts to administrators when events occur. Osiris is compatible with both Windows and Unix systems.
Another question asked is “How do I ensure file integrity checking on network devices, and if the configuration has changed without knowledge of the organization’s administrators?” As discussed previously, monitoring privilege 15 or administrator/console commands in network devices is definitely one way of recognizing configuration changes within network devices however, configurations can also be changed through a mis-configured SNMP protocol where write functionality is available if the MiBs are known to the attacker. An open source product called Rancid can help determine when these events occur. Rancid has the ability to pull the configuration of the network devices at a configured interval and determine whether or not changes have been made to the configuration. If changes have been made, Rancid will email the changes to the network and security teams for investigation. Integration with Rancid and syslog applications is yet to be determined, but is probably in the roadmap for many syslog application products.
Useful Knowledge for Managing and Implementing Syslog
As discussed in this paper, there are many useful skills needed in order to determine a good syslog program. The most useful is probably knowledge of the environment and business needs. Without this knowledge, the security professional responsible for managing the syslog product may not receive the best return on investment for the implementation and may also cause more problems than may have already existed in the environment. In order to gain this knowledge, the security professional will need to branch out and perform a series of environmental interviews with the various IT departments, more specifically the departments responsible for maintaining the systems and network to discuss the project and different syslog architectures and methodologies available within syslog application products, but also discuss implementation to let the different administrators and managers know the organization’s deployment strategy and determine if any logging is missed that the administrators or managers would like. This will not only help the security professional determine new and ongoing issues that these departments are facing and partner with them on receiving budget to alleviate these issues, but also determine the best product to ensure that these issues do not add too much more complexity to these issues by the syslog implementation. The security professional would also need to interview executives and receive legal and executive information to discuss regulatory compliance information and benefits of how a good syslog program can benefit the security defense posture for the organization. Some ideas for this have already been discussed.
The security professional should also evaluate his or her skillset and the skillsets of the rest of the security team. Syslog application products often require basic and/or advanced scripting knowledge and knowledge of regular expressions for a good syslog implementation in order to get the greatest ROI possible. Knowing this skillset and the information from IT and Executives can help the organization determine the best product available within the allotted budget for implementation. It may be determined that the security team has minimal knowledge of scripting, such as perl, and regular expressions and may need to hire someone, utilize an individual from other departments, receive training, or hire the vendor to implement the product on their behalf. In some cases, it may be determined that the organization’s visions of the syslog implementation may be more than they know how to implement and may have to make a decision to not implement some of the strategy immediately and re-evaluate. This still provides a good syslog security posture and a good return on investment for the organization even though not all ideas were implemented immediately.
With some products, it is also good for the security professional or someone on the security department to have a decent knowledge of database applications. Some syslog products store the log data in backend databases which require a working knowledge of database management and archiving procedures. Sometimes have a working knowledge of database management is required because the syslog application will provide an interface that allows the syslog administrator to set an index query or database view of the data that it will query from in order to speed up queries within the system. Without a working knowledge of databases, these databases can get out of control as far as performance of queries, space, and management of the logs in general. If this occurs, the backup strategy and long term cost also increases which can become burdensome, reduce the return on investment, and potentially weaken the posture of the syslog program which in turn decreases the posture of the syslog defense program.
Syslog Product Comparison
Many good syslog applications exist in the world today and each has their own nuances and niche in the marketplace. Some of these products have been mentioned in this paper and might be good to discuss.
OSSEC is an open source syslog project that is commercially supported by Trend Micro, Inc, and also has an online user forum available. Interesting enough, OSSEC also has a user group available on LinkedIn which is administered by the lead developer of OSSEC, Daniel Cid. The documentation for implementing OSSEC is quite straight forward, and the implementation is easy as well. OSSEC also provides some rules already available for use in the product.
OSSEC primarily supports a client/server architecture where each client and server communication is encrypted by a key provided by and created on the server. The server runs on a Unix/Linux based system, but does have clients for Windows for cross compatibility of the systems. Additionally, OSSEC also provides the ability for network devices to push logs to the OSSEC server for filtering at the server level. The management of OSSEC is primarily command-line driven, but does provide a web based GUI for viewing logs, but GUI management is not currently available. OSSEC also provides file integrity checking and alerting functionality. Knowledge of regular expressions and scripting could be helpful in providing a full syslog application deployment because it does have the ability to provide implementation of most of the ideas mentioned. This project is still very active, and very much looked at for integration into current commercially supported syslog products. AlienVault has implemented OSSEC into its OSSIM product.
CA Audit/eTrust Security Command Center is a commercial player in the syslog market. This product also utilizes a client/server architecture and again the client and server communication is encrypted by a key created on the server, and will also provide the ability to push network device logs to it while filtering at the server level. CA Audit server primarily runs on a Windows based system with a SQL server backend. The installation for CA Audit will build and create the necessary SQL databases for the security administrator, so much of the database guess work is already done however, knowledge of the SA password for SQL is required for installation which could bring up some security concerns.
CA Audit is very flexible in that it has a wide variety of client plugins. It supports everything from various Unix flavors, Linux, Windows, various database clients, and has plugins for different network device technologies. CA Audit even provides the ability to manage SMIT logs for those organizations that are required to audit and log SMIT commands. The management interface is also very good and very flexible. The server contains a client side GUI for real time debug logging. The GUI can also be used for real time reporting in the event that an auditor or investigation is asking for specific time frames of reports.
CA Audit also provides a very powerful web interface through eTrust Security Command Center which is often offered with or as an add-on to the CA Audit package. This interface has the ability to create indexes and manage the CA Audit log database, create profiles for specific users to see specific log data that might be interesting to them and can also provide error debugging dumps. The ajax application is very well thought out, manageable, and friendly once the administrator has figured out how to use the system. Things to note about CA Audit, is that the field names are not the standard field names that would be expected by specific Operating Systems. For example, what most people know as Event ID in Microsoft might be mapped to Native ID in CA Audit because Event ID is reserved for the syslog application. Because of this, CA Audit also maintains a vast knowledge of regular expression and a proprietary scripting language that is very powerful and very useful to its system and with the proper knowledge of the system can perform almost every idea mentioned. CA Audit does not have a built-in file integrity checker, but with as powerful as the scripting language is, might have the ability to script something that could provide some kind of integrity information. CA Audit would need to be involved in order to make many of these events a reality however. The ability to monitor multiple files within the same client is also available, but does require knowledge of the application structure and some command-line. This product can be implemented without paying the vendor to implement, however can get difficult without their help. In short, CA Audit is a very good and solid product as well. The support team is excellent and very helpful, but the product can be quite expensive and requires a specialized knowledge of the CA Audit product and query language for filtering and alerting purposes. CA has come out with an upgrade product of CA Audit called CA Enterprise Log Manager that should be just as good if not better than CA Audit.
Verizon provides a syslog hardware device that the company calls a LEC, or Log Event Correlator. This device supports a network based architecture which means that all interesting logs have to be pushed to the device and the logs will be filtered at the device, similar to that of a Cisco MARS device. Management of the device is also performed through the vendor. Verizon provides a web based management interface and a toll free SOC (Security Operations Center) number so that the security professional or professionals responsible for managing the vendor can view logs, build custom on the fly reports, and create new or change existing filters in order to get the biggest benefit of the solution. This type of solution nearly always provides alerting thresholds however, one benefit to using a solution like this is that thresholds and alerting platforms can be set so that if specific possible attack events are noticed within the specified log and filter parameters, the Security Operations Center will pull the escalation file and will begin calling the list of people provided by the organization to notify them of the possible attack.
Often times organizations will also use the same provider for IDS/IPS monitoring which allow the SOC to perform the cross correlation of events between the syslog and IDS to determine help determine whether or not the attack is authentic or a false positive which can provide better escalation results. Other organizations such as Symantec provide these services and technology as well and performs the job only as well as the security professional manages and customizes the service. These services are also good and have their place in the industry. Although they are not exactly necessary, auditors do like the idea of an external vendor managing the device because it means that a contract is in place for someone to constantly be monitoring the logs and it reduces the risk of log tampering on the device itself by someone within the organization however, does provide potential for a drastic increase in network resources. These services and products can also be very expensive, but are typically only as solid as the security professionals that are managing the vendors. If the security professional is not knowledgeable of what to look for in a solid syslog program, these services are only useful in telling auditors that a syslog program exists which becomes the only return on investment.
In conclusion, a solid syslog program can be very beneficial to the security defense posture of any organization. Taking the time to invest and implement some of the ideas such as application monitoring, database monitoring, log tampering, etc., can result in a powerful syslog deployment which can not only meet the obligations of any regulatory compliance standards and increase the security defense posture of any organization, but also enhance an organization’s ability to investigate incident response events, improve stability, and potentially increase productivity to provide full return on the organization’s investment.
Works Cited
Google. Defininitions of Syslog on the Web.
OSSEC. Getting started with OSSEC. 2008-2010. Trend Micro, Inc.
< http://www.ossec.net/main/getting-started-with-ossec/>
OSU Archives & Records Management Handbook-Definitions. Archives & Records Management
Handbook. 2009. Oregon State University.
<http://osulibrary.oregonstate.edu/archives/handbook/definitions/>
PC Magazine Encyclopedia. Syslog Protocol Definition. 1996-2010. Ziff David, Inc.
< http://www.pcmag.com/encyclopedia_term/0,2542,t=syslog+protocol&i=52374,00.asp>
Wikipedia-the free encyclopedia. Internet Bot. 5. September. 2010. Wikipedia.
< http://en.wikipedia.org/wiki/Internet_bot>
Snort(Software). 2. September. 2010. Wikipedia.
< http://en.wikipedia.org/wiki/Snort_(software)>
Syslog. 23. August. 2010. Wikipedia.
< http://en.wikipedia.org/wiki/Syslog>