Log Analysis Cheatsheet¶
This cheatsheet focuses on command-line tools often used in log analysis and security investigations. The commands are presented in ways that allow you to combine them for more efficient analysis.
¶
1. grep + awk (Search and Extract Data)¶
-
Find error messages and print the timestamp (assumes logs have timestamps in the first column):
grep "error" logs.txt | awk '{print $1, $2, $3}' -
Filter logs for a specific pattern, then extract the 3rd and 5th columns:
grep "login" logs.txt | awk '{print $3, $5}' -
Search for lines that match "failed" in one file and extract user information from another file:
grep "failed" logs.txt | awk '{print $2}' | while read user; do grep "$user" /etc/passwd; done
2. awk + sort (Summarize and Sort Data)¶
-
Sum all login attempts (assuming the 1st column is the IP address and the 4th is the number of attempts):
awk '{s+=$4} END {print s}' login_attempts.txt -
Sort failed login attempts by IP address:
awk '/failed/ {print $1}' logs.txt | sort | uniq -c | sort -nr -
Display top 10 IP addresses based on the number of login attempts:
awk '{print $1}' logs.txt | sort | uniq -c | sort -nr | head -n 10
3. grep + sort (Search and Sort Results)¶
-
Find the top 10 most common error types (assuming error types are in the 3rd column):
grep "error" logs.txt | awk '{print $3}' | sort | uniq -c | sort -nr | head -n 10 -
Search logs for IP addresses, then sort them by frequency of access:
grep -oP '\d+\.\d+\.\d+\.\d+' logs.txt | sort | uniq -c | sort -nr
4. awk + cut (Extract and Format Data)¶
-
Display a user’s login history (assuming username is in the 3rd column):
awk '{print $3}' login_attempts.log | sort | uniq -c -
Extract and display specific fields, then format:
awk '{print $1, $3, $5}' logs.txt | cut -d ' ' -f 1,3 -
Find and format IPs, then show number of accesses:
awk '{print $1}' logs.txt | sort | uniq -c | awk '{printf "%-15s %s\n", $2, $1}'
5. grep + sed (Search, Modify, and Filter Logs)¶
-
Find all lines with "login", and replace "failed" with "unsuccessful":
grep "login" logs.txt | sed 's/failed/unsuccessful/' -
Search logs for specific keywords, then highlight the matches:
grep --color=always "error" logs.txt | sed 's/error/\x1b[31m&\x1b[0m/'
6. sort + uniq (Counting Occurrences)¶
-
Count occurrences of specific error codes (assuming the error code is in the 2nd column):
awk '{print $2}' error_logs.txt | sort | uniq -c | sort -nr -
Find the most frequent IP addresses involved in failed login attempts:
grep "failed" logs.txt | awk '{print $1}' | sort | uniq -c | sort -nr | head -n 10
7. awk + grep + sort (Data Filtering and Aggregation)¶
-
Extract and display IPs with a specific pattern, then sort by frequency:
grep "failed" logs.txt | awk '{print $1}' | sort | uniq -c | sort -nr -
Extract entries where the status code is 404, sort by the number of occurrences:
awk '$9 == "404" {print $1}' access.log | sort | uniq -c | sort -nr
8. grep + head (Limit Results)¶
-
Get the first 10 lines with a specific keyword (e.g., "error"):
grep "error" logs.txt | head -n 10 -
Show top 10 most recent login attempts by IP address:
grep "login" logs.txt | tail -n 10 | awk '{print $1}' | sort | uniq -c | sort -nr
9. find + grep (Search Through Logs)¶
-
Search for logs containing "error" across multiple files:
find /var/log/ -type f -name "*.log" -exec grep -H "error" {} \; -
Find all log files modified in the last 24 hours, then search for a specific pattern:
find /var/log/ -mtime -1 -type f -exec grep "suspicious_activity" {} \;
10. tail + grep (Monitor Logs in Real-Time)¶
-
Monitor a log file for real-time login failures:
tail -f /var/log/auth.log | grep "failed" -
Follow logs and only show lines with "error":
tail -f logs.txt | grep "error"
11. cut + sort (Quick Field Extraction and Sorting)¶
- Extract and count occurrences of a field (e.g., IP addresses in the 1st column):
cut -d ' ' -f 1 logs.txt | sort | uniq -c | sort -nr
12. Combining Commands for Advanced Filtering and Analysis¶
-
Get the top 5 IP addresses with failed logins:
grep "failed" logs.txt | awk '{print $1}' | sort | uniq -c | sort -nr | head -n 5 -
Show all unique error codes, formatted with counts:
grep "error" logs.txt | awk '{print $2}' | sort | uniq -c | sort -nr -
Extract and summarize event timestamps, showing the most frequent timestamps:
awk '{print $1, $2}' logs.txt | sort | uniq -c | sort -nr | head -n 10
Windows Process IDs¶
Ah, I see! You're referring to Windows Event IDs, which are logged by the Windows Event Log system. These Event IDs are critical for incident response and cybersecurity log analysis as they help to identify specific events that may indicate suspicious or malicious activity.
Here's a table of Windows Event IDs that are often important in log analysis for cybersecurity and incident response:
| Event ID | Description | Significance |
|---|---|---|
| 4624 | Successful logon | Indicates a successful user logon. Investigate suspicious accounts or logon times. |
| 4625 | Failed logon | Indicates failed logon attempts, which can indicate brute-force attacks or unauthorized access attempts. |
| 4648 | A logon was attempted using explicit credentials | Important for identifying lateral movement or pass-the-hash attacks. |
| 4688 | A new process has been created | Useful to track execution of new processes, including potentially malicious or unauthorized ones. |
| 4689 | A process has exited | Helps to track the end of a process, which can be useful for identifying abnormal behavior or process lifecycles. |
| 4670 | Permissions on an object were changed | Indicates changes to file, folder, or registry permissions, which can be a sign of tampering or escalation of privileges. |
| 4732 | A member was added to a security-enabled local group | Shows when a user is added to a privileged group, which could indicate privilege escalation. |
| 4733 | A member was removed from a security-enabled local group | Could indicate removal of a user from a privileged group (useful in investigating insider threats). |
| 4740 | A user account was locked out | Can be an indicator of brute-force attacks or malicious user behavior attempting to guess passwords. |
| 4767 | A user account was unlocked | Indicates the unlocking of a user account; relevant for identifying account compromises. |
| 5156 | The Windows Filtering Platform has allowed a connection | Shows that a network connection has been established, useful for tracking legitimate or suspicious network activity. |
| 5158 | The Windows Filtering Platform has blocked a connection | Identifies blocked network connections, helpful in spotting attempted communications with external systems. |
| 4688 | A new process has been created (often used for identifying malicious binaries or suspicious scripts) | Identifies the launch of new processes and can help track execution of malware or unauthorized scripts. |
| 4689 | A process has exited | Used for tracking the termination of processes, important for analyzing malware lifecycles or fileless attacks. |
| 5140 | A network share object was accessed | Important for monitoring access to shared resources; potential sign of data exfiltration or unauthorized file access. |
| 5145 | A network share object was accessed with an incorrect password or by unauthorized user | Could indicate unauthorized access attempts or malicious activities attempting to access shared resources. |
| 1102 | The audit log was cleared | A red flag; clearing event logs is often an attempt to cover tracks after malicious activity. |
| 5152 | The Windows Filtering Platform has blocked a connection due to a rule match | Could be useful for detecting attacks blocked by firewall rules (e.g., port scans, exploit attempts). |
| 4680 | A handle to an object was requested by a process | Shows when an object is opened, useful for detecting suspicious file or registry access. |
| 4690 | A handle to an object was closed by a process | Can be used to track whether potentially malicious processes are interacting with system resources. |
| 4698 | A scheduled task was created | Important for identifying unauthorized tasks scheduled by attackers for persistence. |
| 4700 | A scheduled task was deleted | Useful for identifying the removal of tasks created by attackers for persistence. |
| 4769 | A Kerberos authentication ticket was requested | Important for monitoring Kerberos authentication activity; may help identify unauthorized access. |
| 4771 | Kerberos pre-authentication failed | Indicates failed Kerberos authentication; could suggest attacks such as Pass-the-Ticket or credential dumping. |
| 4776 | The domain controller attempted to validate the credentials of an account | Tracks failed login attempts in Active Directory, often associated with unauthorized login attempts. |
| 4000 | DNS query received | Can help in identifying suspicious domain lookups or attempts to reach malicious IPs. |
| 4662 | An operation was performed on an object (e.g., a file, registry key) | Identifies when a change was made to an object, such as file modification, which could be a sign of data manipulation. |
| 4964 | Special group membership was enumerated | Often used to track changes in user group memberships, useful for detecting unauthorized privilege escalation. |
Got it! I understand now. You want to see grep, awk, sed, and other commands being used together in real-life scenarios for log analysis, especially involving Windows Event IDs.
Here’s a more integrated approach where commands are combined to process logs and extract meaningful information:
Windows Logs with Event IDs¶
1. Search for Failed Logon Attempts (Event ID 4625) and Show IPs Using grep, awk, and sed¶
-
Scenario: You're analyzing logs for failed logon attempts (Event ID 4625) and want to identify the source IP addresses.
grep "4625" path/to/log | awk '{print $11}' | sort | uniq -c | sed 's/^ *//g'Explanation: -
grep "4625" path/to/log: Filters logs for failed logon attempts (Event ID 4625). -awk '{print $11}': Extracts the 11th column, which typically contains the IP address. -sort | uniq -c: Sorts the IP addresses and counts unique occurrences (identifying multiple failed logins from the same IP). -sed 's/^ *//g': Removes leading spaces for a cleaner output.Outcome: Displays the number of failed logon attempts per IP address.
2. Check for Multiple Event IDs (4624, 4625, 4648) and Display Usernames Using grep, awk, and sed¶
-
Scenario: You're looking for both successful (4624) and failed (4625) logons, and also logons using explicit credentials (4648). You want to display the usernames involved.
grep -E "4624|4625|4648" path/to/log | awk '{print $5}' | sort | uniq -c | sed 's/^ *//g'Explanation: -
grep -E "4624|4625|4648" path/to/log: Filters for successful logons (4624), failed logons (4625), and explicit credential logons (4648). -awk '{print $5}': Extracts the 5th column, which typically contains the username. -sort | uniq -c: Sorts the usernames and counts occurrences. -sed 's/^ *//g': Cleans up any extra spaces before displaying the output.Outcome: Shows a count of successful, failed, and explicit credential logons by username.
3. Extract Process Creation Events (Event ID 4688) for Specific User Using grep, awk, and sed¶
-
Scenario: You want to track new processes created (Event ID 4688) by a particular user (e.g.,
Administrator).grep "4688" path/to/log | grep "Administrator" | awk '{print $0}' | sed 's/^/NEW PROCESS: /'Explanation: -
grep "4688" path/to/log: Filters for process creation events (Event ID 4688). -grep "Administrator": Filters those events further to include only those related to the Administrator user. -awk '{print $0}': Outputs the full line for each matching event. -sed 's/^/NEW PROCESS: /': Adds a prefix to each line for better readability and context.Outcome: Displays process creation events for the Administrator user, making it easier to see what processes were launched.
4. Monitor Logon Events and Check for Concurrent Logons (Event ID 4624) Using grep, awk, sort, and uniq¶
-
Scenario: You are analyzing successful logon events (Event ID 4624) and want to identify any concurrent logons by the same user.
grep "4624" path/to/log | awk '{print $5, $1}' | sort | uniq -dExplanation: -
grep "4624" path/to/log: Filters for successful logon events (Event ID 4624). -awk '{print $5, $1}': Extracts both the username (5th column) and timestamp (1st column). -sort: Sorts the output by both username and timestamp. -uniq -d: Displays only the duplicate (concurrent) logons by the same user.Outcome: Identifies concurrent logins by the same user across different machines or sessions, which can indicate suspicious behavior.
5. Search for Logon Events with Specific IP Addresses Using grep, awk, sort, and sed¶
-
Scenario: You want to filter successful logon events (Event ID 4624) and find logons from a specific IP address.
grep "4624" path/to/log | awk '{if ($11 == "192.168.1.100") print $0}' | sed 's/^/IP LOGON: /'Explanation: -
grep "4624" path/to/log: Filters for successful logon events (Event ID 4624). -awk '{if ($11 == "192.168.1.100") print $0}': Filters out the lines where the IP address (11th column) matches 192.168.1.100. -sed 's/^/IP LOGON: /': Adds a prefix to each matching log entry for clarity.Outcome: Displays only successful logon events from the specific IP 192.168.1.100, helping to identify a specific source of access.
6. Monitor Event ID 1102 (Audit Log Cleared) and Extract Related Events Using grep and awk¶
-
Scenario: You want to track when the audit log is cleared (Event ID 1102) and then look at subsequent events for further investigation.
grep "1102" path/to/log | awk '{print $0}' | sed 's/^/LOG CLEARED: /' > cleared_logs.txt grep -f cleared_logs.txt path/to/logExplanation: -
grep "1102" path/to/log: Finds when the audit log is cleared (Event ID 1102). -awk '{print $0}': Prints the full line for each matching event. -sed 's/^/LOG CLEARED: /': Adds a prefix to identify log-clearing events. -> cleared_logs.txt: Saves the filtered log-clearing events to a file. -grep -f cleared_logs.txt path/to/log: Uses the saved file of cleared logs to find subsequent events that may have been tampered with.Outcome: Tracks log clearing events and then allows you to follow up with any suspicious activity after the logs were cleared.
7. Combine and Filter Multiple Event IDs (4624, 4688, 4689) for User Activity Using grep, awk, and sort¶
-
Scenario: You're analyzing user activity and want to check for successful logons, process creation, and process termination.
grep -E "4624|4688|4689" path/to/log | awk '{print $5, $1, $0}' | sort | uniqExplanation: -
grep -E "4624|4688|4689" path/to/log: Filters logs for successful logons (4624), process creation (4688), and process termination (4689). -awk '{print $5, $1, $0}': Extracts username (5th column), timestamp (1st column), and the full log entry. -sort | uniq: Sorts the logs and removes any duplicates.Outcome: Provides a chronological sequence of user logins, processes created, and processes terminated, helping you analyze user activity.