Slay the Log4Shell Dragon TEAM 2 - Hunt and Detect Attacks Playbook
Slay the Log4Shell Dragon Playbook TEAM 2 – Hunt and Respond Playbook
Struggling with how to tackle the Log4J / Log4Shell Dragon and low on resources?
First, as I've said before, start with my “RAPID LOG4SHELL RESPONSE 1-PAGE CHECKLIST” to begin immediate actions to tackle this issue. In that 1-pager, I provide concise guidance to get started quickly.
However, for medium and larger sized companies, this approach might not be enough, although it is a great and immediate start.
Simply patching alone will likely not meet the expectations in the case of the Log4J / Log4Shell vulnerabilities, if there is a breach. A more comprehensive approach is required to reduce risk. I’ve therefore put this playbook together to help go above and beyond just a patching-based approach and hope it proves useful.
My goal was to provide something that would help ensure a high enough level of due diligence for risk reduction of this issue.
In this article I’ll provide a TEAM 2 PLAYBOOK of recommended tasks and guidance
To Reiterate what I stated in Article 2A - Here’s what I recommend to respond to this critical vulnerability:
INITIAL STEP – Quickly put together two high-performance teams or HPTs. One team should focus on security engineering related to the issue to enable exploit protection and detection. The second team should focus on compromise detection via hunt and response actions.
Both teams should collaborate closely, but their focus is different and should be worked in parallel. Here’s what I recommend for these two teams and task areas or lines of ops, to be done simultaneously:
Team/Task Area 1 for immediate protective and detective security engineering
- Mission: Protect, instrument and defend the enterprise quickly from Log4J vulnerabilities
- Goal: Complete tasks within 30 days and then transition to longer-term process
Team/Task Area 2 for immediate compromise detection hunt actions and attack response
- Mission: Hunt for signs of Log4J compromise and quickly respond, share, defend
- Goal: Complete tasks within 30 days and then transition to longer-term process
Purpose of this playbook: to provide recommended Immediate Actions and Response Playbook to answer the question: “What should be done ASAP to Defend Against the Log4Shell Dragon? This will be done for the above two teams and lines of effort. Also note I released a Log4J vulnerability scan tool compiled list and advice.
A Team Effort and Continued Defense-in-depth
Again, this issue requires a dedicated line of effort, commitment and teamwork to reduce risk. I recommend to activate the Cyber Incident Response Team or CIRT/CERT process to initially tackle the issue with immediate actions, if no other process exists to deal with this initially.
The team should follow established "incident response" procedures (and/or modified as needed) for tackling this in an emergency way, to the best of their ability. This strategy will be based on defense-in-depth continuing strategy, as this is still needed to address cyber risk.
I feel the sequence provided should help provide immediate risk reductions progressively. That said, each environment is different, so I strongly encourage you to determine what works best for your context regarding risk
Note: this second playbook would not have been possible without the generous contributions of the entire cybersecurity community. My hat is off to them for quickly developing many great resources in a short time. Many of these great resources are linked herein, reflecting that this is a team effort at the end of the day. I hope this will provide some efficiencies to help others avoid burnout and help make the efforts more manageable and frictionless.
Caveats: Links and advice at links or tools have not necessarily vetted - Quality of resources not known nor tested – visit links and/or follow advice herein completely at your own risk.
==============================================================
TEAM 2 - IMMEDIATE ACTIONS TO HUNT FOR AND DETECT ATTACKS
===============================================================
OUTLINE
#1 – REACH OUT TO YOUR SECURITY MSSPs and PRODUCT VENDORS and PARTNERS
#2 - CHECK FOR INDICATORS OF COMPROMISE / ATTACK (IOCs/IOAs) with EXISTING TOOLS
#3 - CHECK FOR IOCs/IOAs USING WIDELY RECOMMENDED TOOLS from KNOWN SOURCES
#4 - PROVIDE FEEDBACK TO TEAM 1 AND PREPARE TO TRANSITION TO THE TIGER TEAM
======================================================================
RESPONSIBLE TEAM – LEADS - POINTS OF CONTACT – REPORTING - TIMELINES
======================================================================
Divide and conquer on all of these major task items. Assign each task to a responsible are by certain deadline. Example and template below:
#1 – REACH OUT TO YOUR SECURITY MSSPs and PRODUCT VENDORS and PARTNERS
Who: TBD
Reporting Cycle: TBD
Responsible Leader: TBD
By When: TBD
#2 - CHECK FOR INDICATORS OF COMPROMISE / ATTACK (IOCs/IOAs) with EXISTING TOOLS
Who: TBD
Reporting Cycle: TBD
Responsible Leader: TBD
By When: TBD
#3 - CHECK FOR IOCs/IOAs USING WIDELY RECOMMENDED TOOLS from KNOWN SOURCES
Who: TBD
Reporting Cycle: TBD
Responsible Leader: TBD
By When: TBD
#4 - PROVIDE FEEDBACK TO TEAM 1 AND PREPARE TO TRANSITION TO THE TIGER TEAM
Who: TBD
Reporting Cycle: TBD
Responsible Leader: TBD
By When: TBD
Log4Shell Metric Group 3 (Team 2 Responsibility) – All security, monitoring and SIEM tools and services (aka “cyber capabilities”) logs were checked and no evidence was found of compromise or exploit in any:
- A. External-facing devices/systems
- B. Internal-facing devices/systems
- C. Third parties known at this time (using proactive approach and deliberate statements and verifications)
You may want to list them as follows for accountability and easy visibility to CISO / CIO:
- Nothing found
- Possible attack - investigate further
- Possible false positive - investigate further
- Probable attack - investigate further
- Probable false positive - no investigation
- Confirmed attack - immediate response "high"
- Multiple confirmed attack - immediate response "critical"
=================================================
TEAM 2 RESPONSE - DETAILED INSTRUCTIONS
=================================================
NOTE: IF POSSIBLE - RUN AS MAY OF THESE IN PARALLEL - ESPECIALLY #1 and #2
#1 – REACH OUT TO YOUR SECURITY MSSPs and PRODUCT VENDORS and PARTNERS
- I think this is #1 because there is least friction using existing tools to tackle the issue
- This may first involve going to customer portals first, then calling if the info is not clear
- Ensure security systems are patched and stay up-to-date with the most current
- A quick 30 min or 1 hour call with a vendor is easy and may provide an easy way to get quick guidance and tap into valuable lessons learned that can be shared with you
Managed Security Service Providers aka MSSPs (outsourced/contracted)
CISecurity recommends this regarding MSSPs: “ensure they are aware of, monitoring, and taking appropriate actions on any alerts associated with the presence of Log4j in your environment. Ask them specifically for evidence of outbound traffic as a result of a Log4j exploitation attempt.” Ref: https://www.cisecurity.org/log4j-zero-day-vulnerability-response/
Security Product Vendors
I recommend jumping on a 30 minute call with your NGFW and IPS/IDS/UEBA vendors to:
A. Understand what they’re doing to help reduce your risk
B. Obtain up-to-date detection advice
C. Ensure you have the most up-to-date detection signatures for Log4J for your tools
D. Provide help with any specific requirements or concerns to maximize product ROI
These details should be shared across the blue team and with leadership when appropriate. Ask what they’re incorporating into their products to defend against Log4J and what they have planned as well as what they recommend based on the community.
If you don’t have a “belly button” to help organize and orchestrate these efforts, consider outsourcing them to a trusted resource.
=====================================
#2 - CHECK FOR INDICATORS OF COMPROMISE / ATTACK (IOCs/IOAs) with EXISTING TOOLS to check whether one had been attacked already and possibly compromised. If so, implement prioritized incident response plans across the enterprise as appropriate.
- I feel this is a close second and should probably be worked in parallel with #1 above perhaps by a joint sub-team under team 2 consisting of red and blue team and IT folks that can provide the best and most scalable assistance when looking for compromise
- Again looking for compromise along the entire kill chain must remain a primary function of the cyber and SOC teams – as Log4J is a means to an end not an end in itself – other steps of the kill chain might be used and more easily spotted even if initial exploit and compromise footholds are ambiguous or not detected
Using existing tools first to look for detections:
Teams should still use the existing tools they normally use for overall attack and post-compromise detections, as well as some possible Log4J attacks. Teams should also use other scan tools as well.
Before I provide some recommendations on other tools, here are some recommendations using existing tools as part of the detection strategy:
2-1. OVERALL – EMPLOY EXISTING RESPONSE TO “INTRUSION and POST-EXPLOIT KILL CHAIN”
A. Check all security tools via typical and most efficient means for typical Attack Indicators
- Using IOCs and Kill Chain post-exploitation indicators by using all available tools already in the environment (EDR, NGFW, IPS, IDS, SIEM, Logs, Events, UEBA, XDR etc.)
- Uptick in scanning activity or any activity related to servers especially Apache is a string possible indicator
- Lateral movement indicators, etc.
- Respond using normal incident response processes
- Try to determine attack vectors and fix those first
- See also next steps to facilitate spotting attacks specifically related to Log4J
B. Per TrustedSec, perform the following analysis on endpoints (especially servers), network
Endpoint:
- “many EDR and endpoint security systems have implemented signatures to detect this activity.
- Suspicious execution of common command line tools used to download files such as: curl, wget, or PowerShell
- The creation of suspicious or unexpected programs or services on an endpoint
- An increase in CPU and memory usage on a server (This is due to many attackers placing cryptominers on exploited systems.)
- Endpoint security software generating alerts for post-exploitation tool usage or activity”
Network:
- “Once an attack is successful, the victim system will often reach out to the attacker’s staging server to download malicious code. Netflow data can significantly assist with the identification of this communication at the network layer or by utilizing EDR telemetry at the endpoint layer. Therefore, the following network analysis can be performed to look for post-exploitation activity:
- Search for outgoing network or web connections from your servers to the Internet.
- Outbound network connections may be to non-standard ports or over standard HTTP/S ports.
- Look for suspicious curl or wget user agents to external IP addresses.
- Examine DNS for queries to suspicious or known malicious sites. A list of known domains and callback URLs can be found at the following links:
- Known scanning IPs – https://gist.github.com/gnremy/c546c7911d5f876f263309d7161a7217
- Known callback URLs – https://gist.github.com/superducktoes/9b742f7b44c71b4a0d19790228ce85d8
- Zeek Log4j Intelligence Feeds – https://github.com/CriticalPathSecurity/Zeek-Intelligence-Feeds/blob/master/log4j_ip.intel”
2-2. CHECK LOG FILES FOR INTRUSION AND ATTACK INDICATORS
A. Infocyte provides some excellent advice regarding how to discover exploit attempts:
[proceed with caution and may want to test in isolated test environment first]
“Luckily, there are a couple ways to detect exploit attempts while monitoring the server and uncover previous exploit attempts:
Review apache logs for jndi:ldap, jndi:rmi or jndi:dns. These are the magic strings that cause the logger to go haywire and follow/execute the url that follows it.
Scan /var/log with yara signatures matching some of these indicators
Scan the webserver for generic webshells
If you have EDR on the web server, monitor for suspicious curl, wget, or related commands. Likely the code they try to run first following exploitation has the system reaching out to the command and control server using built-in utilities like this.
NOTE: If the server is exploited by automated scanners (good guys are running these), it’s possible you could get an indicator of exploitation without follow-on malware or webshells. Some research scanners exploit the vulnerability and have the system send out a single ping or dns request to inform the researcher of who was vulnerable.” [highlights/emphasis mine]
B. Dragos also provides some advice on where to look initially:
“If the host system is Linux-based, you can run one or more of the following commands to detect the presence of Log4j. Note these commands will not be able to search within VMs, Docker containers or Kubernetes containers. You will need to interrogate virtual machines and containers separately.
[proceed with caution and may want to test in isolated test environment first]
dpkg -l | grep liblog4j
dpkg -l | grep log4
find / -name log4j-core-*.jar.
locate log4j|grep -v log4js
On Windows, you can run the following commands to find Log4j:
dir c: /S /b | findstr /C:"log4"
Within Java projects, it is possible to embed a Jar file such as Log4j inside of another Jar file. So, it is also possible the Log4j jar file could exist within a larger Jar file. A Jar file is just a zip file that contains various Java project files.” [highlights mine. I’ll also provide a link to their tool and some others further down]
C. TrustedSec Recommendations for Logs
TrustedSec says to do this to find uncompressed and compressed log files that are:
“Manual searches can also be performed, but the following do not consider any obfuscation”
Utilize the following Linux commands to search uncompressed log files in /var/log. Note that /var/log should be changed to include any additional application-specific directories.
[proceed with caution and may want to test in isolated test environment first]
$ sudo find /var/log/ -type f -exec sh -c "cat {} | sed -e 's/\${lower://'g | tr -d '}' | egrep -i 'jndi:(ldap[s]?|rmi|dns):'" \;
$ sudo egrep -i -r '\$\{jndi:(ldap[s]?|rmi|dns):/[^\n]+' /var/log
Utilize the following Linux commands to search compressed log files in /var/log.
$ sudo find /var/log -name \*.gz -print0 | xargs -0 zgrep -E -i '\$\{jndi:(ldap[s]?|rmi|dns):/[^\n]+'
$ sudo find /var/log/ -name "*.gz" -type f -exec sh -c "zcat {} | sed -e 's/\${lower://'g | tr -d '}' | egrep -i 'jndi:(ldap[s]?|rmi|dns):'" \;
More TrustedSec Log Search Recommendations for Detection of Log4J exploit attacks:
LOG ANALYSIS
Exploitation attempts can be detected by searching through all log files on a server for JNDI-related exploit strings. While straight searches can be performed, attack strings can be easily obfuscated. The following script takes the obfuscation into account when searching:
Log4Shell-detector (https://github.com/Neo23x0/log4shell-detector) – Detector for Log4Shell exploitation attempts]()
“Manual searches can also be performed, but the following do not consider any obfuscation” (Reference – https://gist.github.com/Neo23x0/e4c8b03ff8cdf1fa63b7d15db6e3860b)
Additional regular expressions that can be used to detect attacks can be found on https://gist.github.com/karanlyons/8635587fd4fa5ddb4071cc44bb497ab6
=====================================
#3 - CHECK FOR IOCs/IOAs USING WIDELY RECOMMENDED TOOLS from KNOWN SOURCES
- I made this one #3 because using bringing other tools into the environment (as opposed to using existing security tools or performing queries with existing network and system tools) always requires additional logistics, effort and risk (i.e. friction)
- I have tried to provide what seem to be the most promising and impactful tools up front is a rough sequence but because I have not tested these so I cannot make any guarantees
Proceed with caution at your own risk when it comes to other tools and code. I make no guarantees whatsoever, use at your own risk. Also, go to the end of this article to see my caveats and recommendations to test tools first before using them in an operational environment, get permission and ensure accountability.
If any tools in test environments show anything undesirable then do not use in production. I have not tested any tools and do not know their quality or unintended consequences or full capabilities.
I recommend to run in an isolated and segmented test zone within a test environment first and note any undesired things that may have been included so as to preclude their use in production. This is why I emphasize using existing tools in the environment first.
No tool is perfect, so realize there will be limits for what they can detect, especially when tools are written and pushed out so quickly. Also assume attackers could eventually use tools and attacks not yet covered, so stay up-to-date with new developments.
------------------------
A. Attack Rules to Instrument for Detection (a couple may be repeats but re-listed in case)
Yara Rules to institute to detect JNDI exploit attempts
---------------------------
Sigma Rules to institute to Detect exploitation attempt against log4j RCE vulnerability reported as CVE-2021-44228 (Log4Shell)
Another Sigma Github page per CISA
Rules that Detect exploitation attempt against log4j RCE vulnerability reported as CVE-2021-44228 in different header fields found in web server logs (Log4Shell)
---------------------------
Credit to https://securityblue.team/log4j-hunting-and-indicators/ for many of these resources and visit their site for more IOCs including payload indicators, attack source IP addresses, User-Agent indicators, and other exploit indicators as well as what was most commonly seen. These indicators are great, but also get stale quickly so ensure any sources of IOCs stay up-to-date. If they do not, move on also automate these things because IOCs upkeep is too tedious manually.
Snort and Suricata Emerging Threat Rules: [I expect these will change quickly]
---------------
Payload Indicators: [these will change and could be obfuscated]
---------------
Source IP Indicators: [these will change and could be obfuscated]
---------------
User-Agent Indicators: [these will change and could be obfuscated]
---------------
Log4J Callback Domains: [these will change and could be obfuscated]
---------------
Hunting Tips: (your threat hunt and cyber threat intel team needs to keep up on the latest and greatest recommendations and changes in this area) - [these will change and could be obfuscated]
------------------------
B. Indicator of Compromise (IOC) Detections recommended by Upguard:
IP Addresses: [these will change and could be obfuscated]
Callback Domains: [these will change and could be obfuscated]
Payload Detections: [these will change and could be obfuscated]
Greynoise IOCs [these will change and could be obfuscated]
------------------------
C. Indicator of Compromise (IOC) Detections recommended by CISA:
[these will change and could be obfuscated]
Cisco Talos Blog Threat Advisory: https://blog.talosintelligence.com/2021/12/apache-log4j-rce-vulnerability.html
Florian Roth’s GitHub pages:
- Log4j RCE Exploitation Detection https://gist.github.com/Neo23x0/e4c8b03ff8cdf1fa63b7d15db6e3860b
- log4shell-detector https://github.com/Neo23x0/log4shell-detector
Huntress blog: https://www.huntress.com/blog/rapid-response-critical-rce-vulnerability-is-affecting-java
Microsoft Security Response Center: https://msrc-blog.microsoft.com/2021/12/11/microsofts-response-to-cve-2021-44228-apache-log4j2/
NCC Group: Log4Shell: Reconnaissance and post exploitation network detection https://research.nccgroup.com/2021/12/12/log4shell-reconnaissance-and-post-exploitation-network-detection/
Other Refs for the above: https://www.cisa.gov/uscert/ncas/alerts/aa21-356a and https://gist.github.com/Neo23x0/e4c8b03ff8cdf1fa63b7d15db6e3860b (further up).
NCSC-NL Threat Hunting Resources: (they have other helps at their Github link)
NCSC-NL IOC Resources - Consolidated List
NCSC-NL Detection and Mitigation (work with Team 1 as needed)
Elastic
Microsoft 365 Defender advanced hunting queries
Microsoft's Azure Sentinel IoC List
https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/Sample%20Data/Feeds/Log4j_IOC_List.csv
More Exploits and Exploit Detections can be found here – proceed with caution as some of these are possibly unknown resources:
Again remember that these IOC detections change fast and must stay up-to-date constantly. Best to automate the detections and blocks and/or see if your existing security products and vendors are already doing this.
------------------------
D. Use Breach Attack Detection Tools to detect Log4J Exploitability as Part of Attack Chains
Randori, a BAS or Breach Attack Simulation company (Ref: https://www.randori.com/blog/cve-2021-44228/ ) In fact, Randori stated they are offering a free assessment of Log4J attack surface for organizations:
- Randori Log4J assessment: https://www.randori.com/demo/
- Cymulate, a competitor, also has a Log4J assessment they’re offering: https://cymulate.com/log4j-resilience-assessment (not sure if theirs is free)
Perhaps others are offering the same, and this is a great chance for BAS to take hold as it should as a good approach to check cyber risk in many areas. I believe breach attack simulation is the modern form of exploit protection validation and verification against attacks that can accelerate and improve vulnerability risk management significantly. Log4J is a big test for this.
=====================================
#4 - PROVIDE FEEDBACK TO TEAM 1 AND PREPARE TO TRANSITION TO THE TIGER TEAM
Because Team/Task # 2 is focused on finding exploits and actual attacks/attempts in the environment - as well as hunt for footholds - they must share their findings with 2 major lines of operation:
1. Provide details and assist with incident response for any discovered cyber attacks/issues to the CIRT/CERT Team
- Isolating and remediating any attacked and compromised assets
- Looking at the rest of the kill chain to determine further post-exploit compromise and then working quickly to help eradicate
- Simultaneously working these steps recommended in this response playbook
2. Provide critical intelligence regarding attack detection instrumentation effectiveness to Team 1
- Team 1 and team 2 should work together when able to atomically test the Log4J attack protections and detections put in place to determine their effectiveness and make any adjustments
- This is an opportunity where certain automations and breach attack response tools could be useful
The Incident Response (IR) team should use reliable and trusted scan and test tools made available by existing tools in the environment (vulnerability scanners, Antivirus, network, etc.). I’ve provided the links to the ones that seem to be trusted and helpful, plus freely available for immediate use.
If you have any feedback regarding these tools and recommendations, please let the owner/producer of the tool know directly for a fix. I have NO CONTROL OVER the quality or output nor changes to these tools. Use any tool and advice herein entirely at your own risk.
If you desire, let me know if there is anything else you discover for the benefit of the community that I can share i.e. “lessons learned” that could help others save time, energy and effort and avoid friction where possible.
This is a perfect opportunity for the Red Team side (if the capability exists) to help the Blue Team test the vulnerabilities and defenses related to Log4Shell. The Red team should scan externally for any vulnerable devices first, and work with the blue team to fix any issues. This combined synergy should be used to tackle Log4J soonest.
=====================================
Misc Links and Resources
1. Incident Response Resources
Cybersecurity Incident & Vulnerability Response Playbooks
Technical Approaches to Uncovering and Remediating Malicious Activity
2. Miscellaneous Resources
These have not necessarily been vetted - Quality of resources not known or tested – visit and/or follow advice in any links herein completely at your own risk
=====================================
In the next the article, I ’ll make recommendations for a “bottom-up approach” that consists of a comprehensive response to further find and deal with any discovered vulnerabilities. Stay tuned for the next 2 articles:
- Slay the Log4Shell Dragon 3 – Transition to a Tiger-Team and Bottom-up Strategy
- Slay the Log4Shell Dragon 4 – Focus on Application Security - Tackle Root Cause Issues
Stay Tuned for What’s Next and let me know if this was helpful!
Comments
Post a Comment