Black Hat USA 2024: SOC within the NOC

One of many new integrations this 12 months was with a utility written by one in all our SOC crew members. Each safety skilled — particularly SOC analysts and responders — have a collection of “fast lookup” type instruments that they use to reply numerous questions on artifacts in an investigation. Shodan is a superb instance; “given an IP handle, what companies is that IP offering, and what software program are they utilizing to offer it?”. Paste the IP into Shodan, and you’ll probably get your reply. With the Shodan integration in XDR, you don’t even have to repeat and paste — simply click on the IP after which within the drop down menu, click on the Shodan hyperlink. However there are a whole bunch of such instruments…

Ben took one of many integration server templates that Cisco Safety revealed on Github and modified it to our wants. He then hosted it on his cloud supplier of alternative, added it to our XDR configuration and will add these easy pivots to XDR on the fly. A easy edit of a textual content file on the server and we might bounce from any observable to any new related reference web site that anybody had steered.

Cisco XDR is constructed on the beliefs of an open integration framework, with revealed information fashions, API specs and pattern code out there to be modified or used as examples/tutorials (together with precise tutorials at DevNet). This dedication to extensibility permits for modifications such because the above with out requiring any motion from the XDR improvement or product groups, permitting prospects to tailor XDR to their distinctive wants.

For instance, an IP tried <AndroxGh0st> Scanning Site visitors towards the Registration Server, blocked by the Palo Alto Networks firewall.

Investigation of the IP confirmed: It was identified malicious.

Additionally, the geo location is in RU and identified affiliated domains. With this info, the NOC management authorized the shunning of the IP.

XDR: Asset visibility

By: Ben Greenbaum

Because the Black Hat community evolves, completely different distributors are given alternatives to convey their merchandise into the toolbox. On account of this ongoing biking, we didn’t have entry to the depth of intelligence beforehand offered by deployment of a Meraki wi-fi infrastructure. Nevertheless, as a result of functionality constructed into XDR Asset Insights to add a customized CSV file of property, we have been in a position to simply operationalize identified community topography into investigative and response operations.

One of many distinctive challenges of the Black Hat setting is how completely different it’s from a “regular” buyer’s manufacturing community. We’ve got just a few hundred gadgets whose safety is our main objective, however tens of hundreds of unmanaged (and sometimes hostile) gadgets within the native community which we wish to shield from one another and shield the surface world from. This distinctive association very a lot drives house the worth that an EDR brings to an XDR resolution. With out good endpoint visibility, the problem is way better. The good thing about an open XDR strategy that’s not an evolution of an present EDR providing is that it may be one in all a number of EDRs, however “one” is preferable to none.

Malware Analytics

By: Ben Greenbaum

Cisco Malware Analytics (previously Menace Grid) was once more used because the backend file evaluation and malware detection engine. We supported our companions from CoreLight and Netwitness, with evaluation of information pulled from clear textual content (convention attendee) and choose encrypted periods (crucial infrastructure).

As normal, the information present in clear textual content communications have been a great indicator of what varieties of data could be anticipated to be leaked by a crowd of safety professionals, and this 12 months the end result rated a strong “much less terrible.” Probably spicy content material included numerous PII (names, employers, positions, e mail addresses, and many others.) from Black Hat receipts and some company e mail attachments. 

And whereas Umbrella did alert us to some wandering infections phoning house, we are able to say that at the least no malware was transferred…within the clear.

By: Aditya Sankar

Cisco XDR features a built-in automation functionality referred to as XDR automation. When you have heard of Safety Orchestration Automation Response (SOAR), Cisco XDR has the total suite of SOAR options. That features the power to drag-and-drop prebuilt code blocks in a selected sequence to create a customized workflow, executing arbitrary API calls to function one-click response actions and creating guidelines to set off workflows based mostly off a schedule or another standards.

We’ve got been utilizing XDR Automate at Black Hat for 3 years to enhance the Cisco companies to our joint buyer, Black Hat, and have carried out a wide range of use circumstances. Nevertheless, this has usually required fairly a little bit of time to be taught APIs and create a very customized workflow. With the most recent XDR Automation Trade, the Trade web page is used to seek out, view, set up and uninstall pre-written workflows which were launched or authorized by Cisco engineers and content material suppliers. Workflows authored by the group have handed a fundamental high quality verify and are supported by the Cisco DevNet Group on a best-effort foundation. The Exchanges helps allow collaboration between workflow creators and moreover reduces the time it takes for a person to expertise worth from XDR automation.

Shout out to Ivan Berlinson, who wrote a workflow to tug menace logs from the Palo Alto Networks API and create Incidents in Cisco XDR. Since Ivan was variety sufficient to publish the workflow to the Trade, it was extraordinarily simple to import the workflow and get it operational. Putting in a workflow from the trade is actually like strolling by a configuration wizard. It features a description of what the workflow does, the required targets and variables, in addition to a contact particular person for assist. Here’s what the workflow appears like within the Trade simply earlier than set up.

This workflow requires Automation distant, on-premises digital machine deployed over ESXi to make sure correct connectivity to the Palo Alto Panorama equipment. Shoutout to Matt Vander Horst who helped with the vCenter required to deploy the Automation distant equipment. The Trade prompts the person to offer values for the required variables and choose the suitable on-premises goal.

Then the workflow is put in and scheduled to run each quarter-hour by way of an automation rule.

This workflow makes use of the PAN-OS XML API to question for menace logs at this path <?kind=log&log-type=menace&nlogs=50>. This kicks off a search job. As soon as the search job is finished, the workflow retrieves the outcomes and begins parsing the menace logs. A Cisco Menace Intelligence Mannequin (CTIM) sighting is created for every particular person menace log and grouped collectively by inner host IP. Subsequent, a CTIM indicator with the outline of the menace log and a relationship to the corresponding sighting are each created. Lastly, an incident bundle is created with the sighting, relationship and indicator entities and posted to the XDR API. The workflow has logic in-built to verify for duplicate incidents and present indicators.

Here’s what one of many incidents that was created from this automation workflow appears like in Cisco XDR. This gave us as analysts within the SOC an excellent place to begin for an investigation.

These Palo Alto Community menace logs point out a listing traversal assault that goals to entry information and directories which are saved exterior the net root folder. PAN Firewall alerts on listing traversal and accessing </and many others/passwd> from supply IP 192.168.130.86 on common attendee Wi-Fi to vacation spot IP <104[.]198.XXX.2XX>, which resolves to < yXXXXis[.]social gathering>. This area is marked as suspicious by a number of menace intelligence sources and has a medium threat rating of 72 in Cisco Umbrella. The host then proceeded to obtain information from <file://var/run/secrets and techniques/> host with fundamental authentication within the HTTP POST header. This exercise was then correlated to related classroom exercise, however the host MAC handle was not seen in any school rooms.

The vacation spot IP reveals unknown with XDR menace intelligence, however the area it resolves to appears to be suspicious and it’s hosted within the Russian Federation, as seen within the Umbrella console. Listed below are further particulars offered by the Corelight crew in our energetic Menace Looking Slack chanel: HTTP POST exercise to the vacation spot in query reveals a fundamental authentication token that decodes to <admin:p034BHUSA43op> which does appear to be it’s getting used for Black Hat coaching because it says BHUSA within the password. Nevertheless, this supply host’s MAC handle was not seen in any school rooms, solely on the final Wi-Fi.

We did discover the host making related queries like <uri = /token$/ uri=/kubernetes/>, which have been seen within the Superior Infrastructure Hacking class, however it’s not sufficient to attribute this exercise to a category. Anyhow, this habits usually shouldn’t be seen on the final Wi-Fi. On this situation, we didn’t take any motion of blocking the vacation spot IP or forcing a captive portal for host IP because the Black Hat community goals to watch for assault and abuse, however not block malicious site visitors.

Ivan Berlison additionally offered one other workflow to supply an XDR Incident when a file is convicted in Cisco Safe Malware analytics. Corelight, in addition to NetWitness, carve information off the community and submit them to be detonated in Safe Malware Analytics. Here’s what the XDR incident appears like when a file with a menace rating above 90 is seen:

We had an exquisite time innovating and investigating at Black Hat USA 2024! The XDR automation trade was an enormous assist in including extra automation capabilities with very minimal customized work. Take a look at AJ Shipley’s weblog on how utilizing Cisco XDR at Black Hat has accelerated our open ecosystem. We’ll be again once more subsequent 12 months, so lengthy Black Hat!

Splunk Assault Analyzer (SAA)

By: Ryan MacLennan

Splunk Assault Analyzer (SAA) is a brand new addition to our deployment. As you could know, Cisco acquired Splunk this 12 months. Due to this new acquisition we labored with our counterparts in Splunk to get their SAA product provisioned for our use at Black Hat. SAA is a file and URL evaluation platform just like Safe Malware Analytics. SAA makes use of a classy set of standards to find out which engine can be greatest suited to evaluation — like net analyzer, static file evaluation, e mail analyzer, signature engines and/or the sandbox. Whereas the product is able to dynamic and static evaluation, we selected to do solely static evaluation for our use at Black Hat.

What is really highly effective in regards to the evaluation of SAA is its assault chain following functionality: The power to intelligently decide how a human would click on on gadgets in a webpage. It would observe hyperlinks, obtain information and analyze further indicators from community connections, identified malicious information, an unknown malicious file that’s analyzed on the fly, phishing domains and extra. It would observe a logical circulate like a human to find out the trail to compromise. This was attention-grabbing to see in our surroundings because it confirmed the trail from a file, the hyperlinks present in it, to completely different web sites, and every step of the trail had a screenshot for us to observe alongside.

For example, we now have a PDF that was submitted to SAA. It discovered hyperlinks within the file and adopted them to see if they might result in one thing malicious. I’ve blocked out a lot of the URLs, however we are able to see the way it went by the PDF information and clicked on the hyperlinks to seek out out the place it will go.

After SAA did its factor, we might have a look at the file in query and the screenshots that it took. We discovered that this file was the information utilized in a coaching room and every hyperlink was a reference to an article, a coaching useful resource (self-hosted and official), or different informational sources a scholar might have.

We have been in a position so as to add this integration with the assistance of our associate Corelight. We talked to them on day one they usually have been excited to get a brand new integration developed with SAA. Just a few hours later, we had an integration with them. This was a tremendous instance of how all of us come collectively to make the NOC higher at Black Hat yearly.

Umbrella DNS

By: Christian Clasen and Justin Murphy

When you have learn the earlier Black Hat NOC/SOC experiences, you realize that in 2023, we made a change to the DNS design. In prior conferences, we assigned inner forwarders to shoppers by way of DHCP, however didn’t drive their use. Basically, attendees might use any DNS resolvers they selected, and we didn’t intervene. The change we carried out was to start forcibly redirecting DNS site visitors to the on-premises DNS forwarders. You possibly can see within the statistics above that this transformation prompted a major bounce in queries processed by Cisco Umbrella — from 54.4 million to 79.3 million.

The steep enhance in question rely was not surprising. What was surprising, nevertheless, was a lower in question rely between 2023 and 2024. Whereas we don’t know the exact reason for this drop, we do have some theories and methods we are able to check them going ahead.

One doable clarification is the prevalence of encrypted DNS protocols. In recent times, the business has turned its consideration to the privateness, integrity and authenticity issues inherent within the plain-text DNS protocol. To resolve a few of these points, “last-mile” encryption has grow to be a favourite of OS and browser distributors. DNS-over-HTTPS (DoH) and DNS-over-TLS (DoT) are solely a few the most well-liked methods to encrypt DNS between the shopper and the recursive resolver.

Detecting all encrypted DNS could be tough and counting the queries not possible. It’s because TCP is the chosen transport for DoH and DoT, and this permits the shopper to pipeline a number of queries over one long-lived TCP and TLS session. However what we are able to usually discover are the “bootstrap” plain-text DNS queries that allow to shopper to seek out the encrypted DNS server. By working a report in Umbrella for the class “DoH and DoT”, we are able to get a deal with on the most well-liked of those companies:

The entry for <dns.google> is almost certainly indicative of Android cell gadgets who use this DoT resolver by default. The rely of queries for that specific service is prone to be larger as a result of the periods on these gadgets are identified to be short-lived if queries should not frequent sufficient.

On this report, we additionally see “canary” domains comparable to <canary.masks.icloud.com> and <use-application-dns.web>. The latter area is utilized by Firefox to detect when it ought to fall again to unencrypted DNS on the community. Extra particulars on how Umbrella interacts with these is written up within the Umbrella assist article for net browsers and DoH default.

Going ahead, we’ll monitor the statistics of those protocols on the convention networks and see what different info we are able to collect utilizing the total packet seize capabilities of our companions and the menace searching capabilities of Cisco XDR. You possibly can count on this matter to be expanded on within the subsequent convention report.

One of many main causes to at the least monitor DNS is to know tendencies and the way the community at Black Hat is getting used from a excessive stage. There are numerous insights that may be gained from forcing DNS by a centralized service with intelligence. DNS queries exist for locations that host every thing from Malware, Crypto Mining and Phishing to content material classes like Social Media, Finance and Unlawful Actions. Moreover, these domains could be categorized into particular functions as properly. With the App Discovery report in Umbrella, these domains are grouped by software, figuring out the potential use of hundreds of functions. This may very well be net apps or different desktop/cell apps.

As all the time, we proceed to see an increase in app utilization at Black Hat:

• BHUSA 2019: ~3,600
• BHUSA 2021: ~2,600
• BHUSA 2022: ~6,300
• BHUSA 2023: ~7,500
• BHUSA 2024: ~9,300

This 12 months there was one stand out Software Class that has been rising in recognition: Generative AI. It would probably be no shock that there are extra attendees and their instruments utilizing Generative AI. We’ve got gone from seeing it as a footnote in logs to reporting it at RSAC 2024, as we noticed 80 completely different Generative AI instruments getting used.

Examine this to Black Hat 2024, just some months later, the place the entire quantity has jumped to 194.

This doesn’t look like only a distinction in conferences, however fairly a rising pattern and acceptance of those instruments.

Community Assurance

By: Adam Kilgore, Shimei Cridlig, Shannon Wellington and Justin Murphy

The ThousandEyes deployment launched at Black Hat USA 2023 one 12 months in the past. At that convention, we spent many lengthy shifts creating the configurations, design, and procedures that shaped the premise for our convention protection. The deployment was additional improved and streamlined at Black Hat London and Black Hat Asia. At this 12 months’s Black Hat USA 2024, we have been able to broaden our protection considerably whereas persevering with to refine our procedures.

New {hardware}

We added 20 Orange Pi gadgets at Black Hat 2024, along with the 8 Raspberry Pi gadgets we deployed in 2023. We’re nonetheless properly wanting the proverbial thousand eyes, however 28 is much more than 8. We deployed our new fleet of Orange Pi gadgets to watch the wi-fi community, whereas the previous Raspberry Pi gadgets have been used for wired monitoring of Registration, the NOC and core community gadgets.

Orange Pi configuration

Mike Spicer put in loads of time to develop new configuration and deployment procedures for the Orange Pi gadgets earlier than the convention. We have been in a position to make use of a script and a small native community to configure every Orange Pi with a selected SSID and PSK. As soon as the Pi gadgets have been configured and the goal entry factors have been deployed, every Pi was walked to its goal coaching room the place it will robotically connect with the entry level (AP) on bootup and start working its scheduled monitoring exams.

Even with the scripting and automation, the configuration stage nonetheless resulted in a mass of wires (pictured above). Deploying the Pi gadgets resulted in additional strolling than the typical attendee would expertise in a convention (not pictured).

Expanded wi-fi protection

With the extra brokers, we have been in a position to deploy to extra Black Hat coaching rooms. The expanded visibility allowed us to catch extra issues earlier than the coaching rooms went dwell, together with a misconfigured PSK, an SSID that wasn’t broadcasting and an SSID that broadcast however didn’t have web connectivity. We’d like to have an agent for every coaching room for full visibility and validation heading into the convention, however we’re pleased with what we caught and the extra confidence the brokers offered heading into the coaching days.

Because the convention shifted from trainings to the briefing days, we shifted our protection from the biggest coaching periods to massive briefing rooms and heavy-traffic areas like the doorway and Enterprise Corridor. Whereas we nonetheless needed to make powerful strategic selections about what to cowl and what to not cowl, we have been nonetheless in a position to unfold brokers throughout every flooring for common visibility.

Troubleshooting

Our experiences over the previous three conferences had produced well-established troubleshooting procedures and paperwork for the Raspberry Pi gadgets, however the Orange Pi devicess offered recent challenges. We had round 25% of our deployed Orange Pi gadgets require troubleshooting in the course of the first 24 hours after deployment, a regarding price. Log evaluation revealed the wi-fi NIC changing into disconnected and the USB coming into a disconnect loop (the wi-fi NIC is linked by way of USB on the Orange Pi gadgets). The issues with the wi-fi NIC and USB result in a recurring ThousandEyes agent core information — a tough set of issues.

Nevertheless, these points turned out to be remoted fairly than widespread, and by the top of the convention we had a full wi-fi deployment that was staying up all day and in a single day as properly. For what turned out to be remoted wi-fi issues, we developed troubleshooting procedures and documentation.

Automated ticketing

A brand new ticketing system was rolled out at this convention that will create tickets in Slack based mostly on ThousandEyes information or reported points. Beneath is a ticket created based mostly on TE alerts for a selected convention room in the course of the first morning of briefings.

The dashboards in ThousandEyes allowed us to offer fast visible info that confirmed which convention rooms have been experiencing the worst latency, alongside a comparability of latency throughout reporting rooms.

The automated experiences behind every dashboard entry offered extra granular info, together with site visitors path and the latency alongside every leg within the site visitors path.

The brand new ticketing system allowed screenshots like those above to be aggregated within the ticket for crew communication and report protecting.

Troubleshooting WorkflowOn 08/06/2024 at 15:00, we noticed excessive latency to our Inner Umbrella DNS check from the South Seas D Hallway and Enterprise Corridor Brokers. Be aware that the hyperlinks to the investigation views are offered as hyperlinks.

To slender down the view, we used a dashboard filter to give attention to the 2 Brokers.

This confirmed the excessive latency noticed by the 2 Brokers prolonged throughout a number of exams.

From right here, we drilled down on every check to verify the person check outcomes.

Inside this view, we chosen a number of exams working on each Brokers and in contrast the outcomes.

We noticed that there was a latency spike reported by each Brokers.

To grasp the reason for the excessive latency, we drilled all the way down to Path Visualization.

We seen the excessive hyperlink delay between the Agent to its gateway. This means a problem both between the shopper and AP or between the AP and the server room with the router.

To substantiate the reason for the latency, we visited South Seas D. We ran further exams to substantiate that the connection expertise match with the outcomes reported by the Agent. Reviewing the room and topology diagrams additional, we discovered that the AP masking South Seas D was positioned in an adjoining room, and was broadcasting two SSIDs — one for the room it was positioned in, and the opposite for South Seas D. The mixture of the AP placement, the AP servicing two rooms, and the attendee quantity in South Seas D mixed to supply the latency noticed by the Agent. These findings have been shared with the wi-fi crew.

Cellular machine administration at Black Hat: The function of Meraki Programs Supervisor

By: Dalton Ross

The Black Hat cybersecurity occasion in Las Vegas is famend for its cutting-edge expertise and seamless attendee expertise. A crucial part of this success lies in efficient cell machine administration (MDM). Since Black Hat USA 2021, we leveraged Cisco Meraki Programs Supervisor (SM) to deal with a wide range of duties essential to the occasion’s operations. Beneath is an in depth have a look at how the Meraki SM was deployed and the challenges confronted alongside the way in which.

Important roles of cell gadgets at Black Hat

Cellular gadgets have been pivotal in a number of key areas:

1. Registration Kiosk iPad Units (~50 Units): Used at registration kiosks to streamline the attendee check-in course of, the place attendees scan a QR code for immediate badge printing
2. Session Scanning iPad Units (~75 Units): Deployed throughout Black Hat periods to scan registered attendees into every session
3. Lead Retrieval Units (~800 Units): A considerable variety of gadgets have been utilized on the present flooring cubicles to swiftly accumulate sales space customer contact information

Deliberate deployment for Meraki Programs Supervisor

To make sure a clean deployment, our technique included a number of key steps:

1. Pre-State with Apple Automated Machine Enrollment (ADE): Earlier than cargo to the occasion location, all gadgets have been pre-staged utilizing ADE. This allowed gadgets to be configured with a identified SSID for quicker deployment on web site.
2. Segregated Transport: Units have been to be shipped in three distinct groupings, every comparable to one of many roles. This aimed to facilitate swift deployment upon arrival.
3. Dashboard Script for Position Affiliation: A customized dashboard script was ready to leverage the Meraki Dashboard API and simply affiliate enrolled gadgets with their respective roles.
4. Automated Configuration Obtain: As soon as powered up, gadgets have been anticipated to robotically obtain any vital configurations or apps associated to their function, making them prepared for rapid deployment.
5. Well being Monitoring with Cisco ThousandEyes: ThousandEyes brokers have been to be deployed all through the venue to log SM well being at completely different occasion places.
6. Submit-Occasion Manufacturing unit Reset: After the occasion, all gadgets have been to be manufacturing unit erased earlier than being returned.

Challenges and workarounds

As in life, challenges arose that required fast considering and adaptation:

• Software Checklist Adjustments: A final-minute change to the applying checklist for session scanning gadgets was required. Though we initially deliberate to have all configurations prepared beforehand, this surprising change was effectively managed utilizing the Programs Supervisor with just some clicks.
• ThousandEyes Agent Limitations: Since ThousandEyes brokers have been beta SM shoppers, they couldn’t precisely collect connectivity information. This was an anticipated habits, but it surely posed a problem for efficient monitoring. To beat this, NOC members from Cisco ThousandEyes and Cisco Meraki collaborated to hack collectively a proof of idea. By means of arduous work and a number of other iterations, we configured the ThousandEyes brokers to simulate machine check-in site visitors, mimicking legitimate SM shoppers.

Deploying Meraki Programs Supervisor at Black Hat was an intricate however rewarding endeavor. Regardless of going through challenges, our crew demonstrated agility and innovation, making certain the occasion’s operations ran easily. The expertise underscored the significance of flexibility and fast drawback fixing in managing large-scale occasions.

By leveraging superior MDM options like Meraki Programs Supervisor, we have been in a position to present a seamless expertise for attendees and exhibitors alike, showcasing the ability of expertise in occasion administration.

We’re happy with the collaboration of the Cisco crew and the NOC companions. Black Hat Europe will probably be December 9-12, 2024 on the London eXcel Centre.

Acknowledgements

Thanks to the Cisco NOC crew:

• Cisco Safe: Christian Clasen, Matt Vander Horst, Aditya Sankar, Ben Greenbaum, Ryan Maclennan, Adam Kilgore, Shimei Cridlig, Shannon Wellington and Justin Murphy, with distant assist by Jessica (Bair) Oppenheimer
• Meraki Programs Supervisor: Dalton Ross, with distant assist by Paul Fidler and Connor Laughlin. Search for their report on The Meraki Weblog.

Additionally, to our NOC companions:

• NetWitness (particularly Alessandro Zatti)
• Palo Alto Networks (particularly Jason Reverri and James Holland)
• Corelight (particularly Dustin Lee)
• Arista (particularly Jonathan Smith)
• Lumen and your entire Black Hat/Informa Tech workers (particularly Grifter ‘Neil Wyler,’ Bart Stump, Steve Fink, James Pope, Mike Spicer, Sandy Wenzel, Heather Williams, Jess Stafford and Steve Oldenbourg)

About Black Hat

Black Hat is the cybersecurity business’s most established and in-depth safety occasion collection. Based in 1997, these annual, multi-day occasions present attendees with the most recent in cybersecurity analysis, improvement, and tendencies. Pushed by the wants of the group, Black Hat occasions showcase content material straight from the group by Briefings shows, Trainings programs, Summits, and extra. Because the occasion collection the place all profession ranges and tutorial disciplines convene to collaborate, community, and talk about the cybersecurity matters that matter most to them, attendees can discover Black Hat occasions in the USA, Canada, Europe, Center East and Africa and Asia.

Share: