Big Data Security ?

Big Data Security?

With increased customer adoption, there are now hundreds of organizations attempting to capitalize on the “big data” movement. One could make a full time job just out of attending all of the conferences, IEEE, TechCon, Strata, Structure, Government, BioMedicine , to name just a few.

The most popular big data application, Hadoop, began as part of Yahoo’s search engine in 2006 and has since become the primary method for building a warehouse of unstructured data. These data warehouses can start with just a few servers, even running as virtual machines, and they can scale to hundreds of PetaBytes running on thousands of CPU cores.

Batch-oriented analytic applications are best suited for big data solutions, with jobs being scheduled as they can run, from a few minutes to a few days, depending on the amount of data being processed and the processing power of the cluster in use. While the data warehouse was designed originally to process large amounts of public information, the demand is increasing for it to house more sensitive business-critical data, and security is an increasing concern.

There are two aspects to consider when we talk about big data security:

  • How to use big data solutions to process information in a manner that improves the security posture of an organization.
  • How best to protect big data infrastructure, and thus ensure the security of the information contained within.

Security Applications Using Big Data:

More and more organizations are pulling activity data (logs) into Hadoop to better analyze it. This correlation of disparate log data traditionally has been difficult or expensive (or both).

  • The University of Montana now has classes to educate students on cyber security including how to use big data effectively.
  • The IRS uses Hadoop with data from eBay and Facebook combined with banking data to detect tax evasion.
  • NSA pulls assorted surveillance data into a Hadoop data warehouse and processes it to detect suspicious activity.
  • Twitter, Facebook, Netflix, Yahoo, Amazon, and many others use Hadoop to track everything you do.
  • HP Fortify CloudScan and their new HAVEn framework both leverage Hadoop to process large amounts of data and improve an organizations security posture.
  • Enterprises can use the commercial Splunk Hadoop Connect for bidirectional interoperability to a data warehouse.
  • The Open Source project Flume is fast becoming a standard method to pull log data from web servers and other applications into Hadoop where other tools can perform analysis.

Securing Big Data:

Some very sensitive data will end up being processed and potentially available in a shared environment. Traditional security controls such as authorization, encryption, tokenization, and data classification were not built in to Hadoop from the onset. An entire ecosystem of vendors like Vormetric have sprung up to help protect these sensitive big data clusters. There is potentially a higher level of uncertainty here than is seen with cloud technology. Combining cloud technology and big data involves significant potential risk; things can go wrong, and data can be inadvertently exposed to unauthorized access. The following are examples of teams working to secure big data environments:

  • The Cloud Security Alliance (CSA) Big Data Security working group, with leadership from Fujitsu, Verizon, and eBay is researching the areas of encryption, infrastructure, data analysis, taxonomy, governance, and privacy.
  • Organizations that were early users of Hadoop had to make modifications to improve the internal security controls. Accumulo was developed and open sourced by the NSA to manage the way in which data could be accessed.
  • When there is a large quantity of sensitive data in one place, the data must be scrubbed to ensure the data warehouse is storing only the data required to solve current business issues. Failure to do this elevates the risk of exposure or cost of protection. Netflix encountered this while gathering customer data during a 2009 contest. Tokenization is a challenging issue to solve but important to get right.
  • Encryption might seem like a simple solution despite the inherit performance penalty or “security tax” but with large data sets and many stake holders key management becomes an overwhelming concern. Solutions such as High Cloud Security encrypt data at rest and offer the ability to shred data when needed.

Much of Hadoop’s improved security controls will be available out of the box for its upcoming versions; however, many existing applications will need significant effort to be ported over. As adoption of big data increases, a common approach has been to rotate out older compute clusters as new upgraded clusters become available. This causes a lot of churn but does ensure that the newest features with the improved security and the faster processing power are always available. Gone are the days of installing a cluster of thousands of servers and then expecting it to be in use for a few years without interruption. Stay tuned as we cover the latest trends in big data security.

Follow me on Twitter: @iben.


Google Cloud Security Command Center 2023

Setup and evaluation of Google Cloud Security Command Center March 2023

When setting up GCSCC for the first time the following options are presented. By default only the Standard Edition features are available.

The Standard tier includes the following services and features:

  • Security Health Analytics: in the Standard tier, Security Health Analytics provides managed vulnerability assessment scanning for Google Cloud that can automatically detect the highest severity vulnerabilities and misconfigurations for your Google Cloud assets. In the Standard tier, Security Health Analytics includes the following finding types:
    • Dataproc image outdated
    • Legacy authorization enabled
    • MFA not enforced
    • Non org IAM member
    • Open ciscosecure websm port
    • Open directory services port
    • Open firewall
    • Open group IAM member
    • Open RDP port
    • Open SSH port
    • Open Telnet port
    • Public bucket ACL
    • Public Compute image
    • Public dataset
    • Public IP address
    • Public log bucket
    • Public SQL instance
    • SSL not enforced
    • Web UI enabled
  • Web Security Scanner custom scans: in the Standard tier, Web Security Scanner supports custom scans of deployed applications with public URLs and IP addresses that aren’t behind a firewall. Scans are manually configured, managed, and executed for all projects, and support a subset of categories in the OWASP Top Ten.
  • Security Command Center errors: Security Command Center provides detection and remediation guidance for configuration errors that prevent Security Command Center and its services from functioning properly.
  • Continuous Exports feature, which automatically manages the export of new findings to Pub/Sub.
  • Access to integrated Google Cloud services, including the following:
    • Cloud Data Loss Prevention discovers, classifies, and protects sensitive data.
    • Google Cloud Armor protects Google Cloud deployments against threats.
    • Anomaly Detection identifies security anomalies for your projects and virtual machine (VM) instances, like potential leaked credentials and cryptocurrency mining.
  • Integration with BigQuery, which exports findings to BigQuery for analysis.
  • Integration with Forseti Security, the open source security toolkit for Google Cloud, and third-party security information and event management (SIEM) applications.
  • When Security Command Center is activated at the organization level, you can grant users IAM roles at the organization, folder, and project levels.

Grant Permissions

Security Command Center created a service account that doesn’t have Cloud IAM permissions. The account must be granted the required IAM roles in order to scan resources for vulnerabilities, store findings, and detect threats.

Required Roles

  • securitycenter.serviceAgent
  • serviceusage.serviceUsageAdmin
  • cloudfunctions.serviceAgent

Service Account Created

  • service-org-id@security-center-api.iam.gserviceaccount.com

Alternately: grant roles manually (gcloud)

$gcloud organizations add-iam-policy-binding id\
    --member serviceAccount:service-org-id@security-center-api.iam.gserviceaccount.com\
    --role roles/securitycenter.serviceAgent &&\
gcloud organizations add-iam-policy-binding id\
    --member serviceAccount:service-org-id@security-center-api.iam.gserviceaccount.com\
    --role roles/serviceusage.serviceUsageAdmin &&\
gcloud organizations add-iam-policy-binding id\
    --member serviceAccount:service-org-id@security-center-api.iam.gserviceaccount.com\
    --role roles/cloudfunctions.serviceAgent

SCC Integrations

Integrate sumo logic with Google Cloud SCC

https://help.sumologic.com/docs/send-data/hosted-collectors/google-source/google-cloud-platform-source/

Log data for Google Cloud Platform (GCP) services is collected and exposed through the Google Cloud Stackdriver service. You can export, in real time, the data collected by Stackdriver to Google Cloud Pub/Sub. We use this Pub/Sub integration to push logs to Sumo Logic in real time.

The Google Cloud Platform (GCP) Virtual Private Cloud (VPC) provides networking functionality to Compute Engine virtual machine (VM) instances, Kubernetes Engine containers and App Engine Flex. The Sumo Logic App for Google Cloud VPC provides visibility into the activities, traffic, and VPC flow in your GCP. The preconfigured dashboards provide you details on the VPC flows, source and destination IP addresses, ports, protocols, and messages.

Compute Engine VPC Flow Logs. These logs provide information from Compute Engine ​VMs ​for ​network ​operations ​such ​as ​Network ​monitoring, ​forensics, ​real-time security ​analysis ​and ​expense ​optimization.

SCC Services

https://console.cloud.google.com/security/command-center/config/services

Select the services that you want to be enabled by default in Security Command Center. You can change these defaults to limit the services to certain folders or projects using advanced settings. Learn more about services

There may be latency between initial activation of services and the availability of findings. Learn more about latency

Security Health Analytics

Identify common misconfigurations in your environment such as open firewalls and public buckets, and CIS violations. Learn more about Security Health Analytics

Manage Premium enablement

These services are not available for Security Command Center Standard. These services are available for projects that are individually activated with Premium tier or for your organization when it is upgraded to Premium tier.

Web Security Scanner

Uncover common vulnerabilities such as cross-site scripting (XSS) and outdated libraries, that put your web applications at risk. Learn more about Web Security Scanner

Rapid Vulnerability Detection

Automatically scan your networks and web applications for critical vulnerabilities that have a high likelihood of being exploited. Learn more about Rapid Vulnerability Detection

During scans, Rapid Vulnerability Detection performs actions that can negatively impact your production resources, like accessing administrator interfaces and attempting to log into your VMs. As a best practice, scan resources in non-production environments before you deploy them to production. Learn more 

Event Threat Detection

Detect threats to your cloud platform, identities, data, and compute instances in realtime. Learn more about Event Threat Detection

Container Threat Detection

Use kernel-level instrumentation to identify potential compromise of containers, including suspicious binaries. Learn more about Container Threat Detection

Virtual Machine Threat Detection

Analyze Compute Engine instances to identify threats, including cryptomining abuse. Learn more about Virtual Machine Threat Detection

possible probable plausible

Possibility is the condition or fact of being possible. The Latin origins of the word hint at ability. Possibility also refers to something that “could happen”, that is not precluded by the facts, but usually not probable. Impossible denotes that something literally cannot happen or be done.

Probability, or chance, is a way of expressing knowledge or belief that an event will occur or has occurred. In mathematics the concept has been given an exact meaning in probability theory, that is used extensively in such areas of study as mathematics, statistics, finance, gambling, science, and philosophy to draw conclusions about the likelihood of potential events and the underlying mechanics of complex systems.

Plausible deniability is, at root, credible (plausible) ability to deny a fact or allegation, or to deny previous knowledge of a fact. The term most often refers to the denial of blame in (formal or informal) chains of command, where upper rungs quarantine the blame to the lower rungs, and the lower rungs are often inaccessible, meaning confirming responsibility for the action is nearly impossible. In the case that illegal or otherwise disreputable and unpopular activities become public, high-ranking officials may deny any awareness of such act or any connection to the agents used to carry out such acts. It typically implies forethought, such as intentionally setting up the conditions to plausibly avoid responsibility for one’s (future) actions or knowledge.

CSPM SOTU 2022 RSA

I was looking forward to attending the latest RSA Conference in the first week of June 2022. Looking forward to meeting friends and learning about all the new ideas various vendors were showcasing on the exposition floor. This year is the first time in last few years we are back in person after the pandemic. Almost everyone is shaking hands and hugging and actiing like every

Campus Network Design

This is an example network drawing for a typical campus location with main data center hosting hundreds of rack mounted servers (LAN) connected to many buildings in with Long Range fiber optic cables (MAN).

Uplink to the WAN supports BGP with multiple links up to 100 Gbps.

The data center switches are low latency with 10Gbps for servers including VXLAN protocol support to work with VMware NSX SDN network overlays.

At the edge there are thousands of 2.5Gbps ports with Power Over Ethernet using 802.1x EAP-PEAP and EAP-TLS for Certificate based Network Access Control.

Enterprise RADIUS with Guest access is supported for the high speed Wireless Network using 802.11ax (Wi-Fi 6) APs.

Campus Network Design

Using a design from 2018 before Brocade was sold off and broken apart here is a parts list to consider. We are looking for similar solutions from major network suppliers today.

This drawing has the following sections:

  • External WAN connections with multiple 1,10, 40, and 100 gbps uplinks
  • 1 pair of MLXe-4 switches for Border Router cluster
  • 1 pair of PAN PA-7080 firewall HA cluster
  • 1 pair of MLXe-32 switches for Internal Router cluster
  • 1 pair of RUCKUS ICX 7850 switches for core distribution to each area using 100 GbE links
  • 10 pairs of VDX 6740 data center top of rack switches with 10GbE on all ports and multiple 40GbE uplinks
  • 1 pair of RUCKUS ICX 7850 aggregation switches to each using 100 GbE links
  • 10 stacks of RUCKUS ICX 7550 access switches for edge with POE and 2.5GbE on all ports. Stack has from 2 to 12 units.
  • Over 100 WiFi access Points would be needed with multiple gigabit connections.
  • All devices have redundant power supplies and fans with 40gbps QSFP interconnections.
  • Network setup, automation, and monitoring would be considered part of the solution.
  • Controller machines would be deployed with Cloud based or Virtual Machines minimizing dependencies on physical hardware deployment.

PAN FW example OSPF with Active/Active High Availability 

In this scenario, the firewalls are deployed in Active/Active HA. This design supports asymmetric traffic, traffic engineering, and consistent deterministic failover behavior. In testing, this design proved to be highly resilient and fast to recover. This design can tolerate the loss of any two network connections without degrading performance or availability. 

Following is a diagram of what will be implemented: 

From Palo Alto Networks OSPF guide

Set the link costs such that certain routes will be preferred over other routes. The link costs are specified to keep the traffic routing symmetric. This also simplifies troubleshooting, packet captures, and firewall log monitoring. 

Note: Floating IP addresses (“Virtual Address”) are typically used when the firewall is adjacent to end hosts. In this scenario, the firewall is directly connected to routers, so floating IP addresses are not used. 

Configure HA as Active/Active. For details on the meanings of the settings, refer to the following article on Active/Active HA in the Palo Alto Networks Knowledge base: https://live.paloaltonetworks.com/docs/DOC-1765 

Note: The path monitoring and link monitoring configurations are not shown below. Make sure that you configure those appropriately. Refer to the document above for help on configuring those settings. 

Dev Test Prod Multiple Environments for a more Secure SDLC

Introduction

A small team of developers writing software for web based application services might not appreciate the need for having their code published to multiple environments. It does take some extra tooling to setup and this effort might slow down the schedule adding a few days to get that new feature out to the production go live environment.

It’s business as usual for large enterprises with different teams responsible for code, operations, networking, security, etc to have several Dev, Test, Verify, and Prod environments for a more resilient and secure software development lifecycle (SDLC).

As an extreme example of what can go wrong due to a software mistake that makes it to production this is the story of how a company with nearly $400 million in assets went bankrupt in 45-minutes because of a failed deployment.

SDLC Security Environment Release Tool Mapping

SDLC Stages – Feature Promotion Process

Software goes through a number of stages on its way from development to production. Each organization will need to adopt these concepts to their own capabilities and requirements. This drawing shows an example of how features are promoted with code going through various stages of the SDLC from development, testing, verification, integration, and deployment before finally being released into the live production environment. At each stage it is helpful to stand up a running version of the application to facilitate the functional and security testing. Some applications have SLAs that require elaborate performance and load testing to be done in a verification environment that mirrors production. Kubernetes facilitates this as we can simply create an alternate namespace using ACLs for authentication and global load balancer rules to ensure only authorized testers have access.

Microsoft Example from 2010

This is not a new concept. Here’s an article Microsoft published 10 years ago:

At a high level, the application goes through these stages as part of the development and deployment process:

  1. A developer checks some code into Team Foundation Server (TFS).
  2. TFS builds the code and runs any unit tests associated with the team project.
  3. TFS deploys the solution to the test environment.
  4. The developer team verifies and validates the solution in the test environment.
  5. The staging environment administrator performs a “what if” deployment to the staging environment, to establish whether the deployment will cause any problems.
  6. The staging environment administrator performs a live deployment to the staging environment.
  7. The solution undergoes user acceptance testing in the staging environment.
  8. The web deployment packages are manually imported into the production environment.

These stages form part of a continuous development cycle.

Microsoft drawing SDLC 2010

Oracle example 2015

2.1.1 Definition of Development Environment

An Oracle development environment is typically an installation on a single host computer (such as a Microsoft Windows desktop or laptop computer or a Linux computer). The requirements for a development environment are very different from the requirements for a production environment.

There is no need for high availability in a development environment, and the number of components and products that can be installed is typically limited to those required by the software engineer or the application that the software engineer is developing.

2.2.1 Planning for a Production Environment

An Oracle production environment is an installation where the products have been configured to deploy production-ready applications and features to your application users.

Unlike a development environment, a production system typically takes advantage of more advanced features, such as server clusters and is deployed to multiple regions.

Gigaom Key Criteria Report on Vulnerability Management

  • https://gigaom.com/report/key-criteria-for-evaluating-vulnerability-management-tools
  • Vulnerability management tools scan your IT estate to help identify and mitigate security risks and weaknesses. These tools can facilitate the development of a more comprehensive vulnerability management program. Leveraging people, processes, and technologies, successful initiatives effectively identify, classify, prioritize, and remediate security threats.

    A security vulnerability is a weakness that can compromise the confidentiality, integrity, and availability (CIA) of information. Attackers are constantly looking to exploit defects in software code or insecure configurations. Vulnerabilities can exist anywhere in the software stack, from web applications and databases to infrastructure components such as load balancers, firewalls, machine and container images, operating systems, and libraries. This includes code used in the CI/CD pipeline as well as the infrastructure-as-code (IAC) that defines the compute, network, and storage infrastructure.

    Recent cybersecurity events have exposed widespread vulnerabilities involving the exploitation of zero-day malware and unknown weaknesses. Threat actors continually discover new exploitation tactics, techniques, and procedures (TTPs) to take advantage of weaknesses throughout integrated systems. Moreover, identifying breach paths is increasingly complicated due to the widespread adoption of ephemeral services.

    Vulnerability management solutions should provide end-to-end visibility of the protect-surface by aggregating both platform and application risks in a single pane of glass, while leveraging prioritized remediation based on business risk and threat context for efficiency. Containerized workloads deployed via DevOps pipelines have unique security requirements that demand a fully integrated vulnerability assessment to be automated into cloud platform services running containerized workloads.

    The path to a mature security posture starts with the ability to identify vulnerabilities in software code, third-party libraries, and at runtime. In addition, the cloud platform used to host your applications should be scanned for misconfigurations. This requires the use of policy configuration baselines, benchmarks, and compliance standards that apply to both the infrastructure and the code used to build it. As organizations implement security guardrails early in the software development lifecycle (SDLC), they can take advantage of cloud-native culture to ensure network and security tools are used throughout all phases of the SDLC.

    This GigaOm report explores the key criteria and emerging technologies that IT decision makers should evaluate when choosing a vulnerability management solution. The key criteria report, together with the GigaOm radar report that evaluates relevant products, provides a framework to help organizations assess the solutions currently available on the market and how these tools fit with their requirements.

    Links

    Here are some links to review for more info…

    Training Video Portal Concept

    How to setup a portal that allows prospects to browse and purchase videos with training material.

    1. Build a web page with a catalog of videos. This is a starting point. https://github.com/GoogleCloudPlatform/microservices-demo
    2. Allow guests to view short clips of the videos and a description with short text extract and reviews – short versions of the videos are available from MS Stream – https://www.microsoft.com/en-us/microsoft-365/microsoft-stream?rtc=1
    3. Guests select a training video to purchase – they add it to the cart
    4. Upon checkout they go to the stripe for payment processing: https://stripe.com/docs/payments/checkout/client#create-products
    5. Once the purchase is made and email with the link to view the full video is available from Microsoft Stream: https://www.microsoft.com/en-us/microsoft-365/microsoft-stream?rtc=1
    6. A log of all activity will be available for uploading to the remote logging server.

    Troubleshooting Technical Problems

    Have you ever done an Operating System update only to have something break like your wireless networking? Follow these steps to resolve the issue quickly and without a lot of drama. Let’s see how it works in practice since sometimes these things can get complicated. One problem might have a simple fix but sometimes you will discover another issue that makes the solution more complicated.

    Here’s an example following using Cisco’s eight-step troubleshooting method to fix your network:

    1. Define the problem.
    2. Gather detailed information.
    3. Consider probable cause for the failure.
    4. Devise a plan to solve the problem.
    5. Implement the plan.
    6. Observe the results of the implementation.
    7. Repeat the process if the plan does not resolve the problem.
    8. Document the changes made to solve the problem.

    For our example the wireless network wouldn’t get an IP address after the OS upgrade. To resolve this we could simply try to revert back to the previous build. Or we can see if the issue can be resolved correctly by updating the wireless drivers. Here’s the updated plan…

    Troubleshooting Wireless on Acer Laptop after Windows 10 upgrade

    1. Define the problem.
      – wireless network wouldn’t get an IP address after the OS upgrade
    2. Gather detailed information.
      – Windows 10 Insider Build 21292 rs prerelease 210108-1514
      – Laptop is Acer Aspire A715-71G
    3. Consider probable cause for the failure.
      – drivers need to be updated for new OS install
    4. Devise a plan to solve the problem.
      – download new drivers from Acer: https://www.acer.com/ac/en/US/content/support-product/7296?b=1&pn=NX.GP8SV.005
      – since our networking isn’t working we need to try Ethernet cable to download since wireless is down
      – install new drivers
    5. Implement the plan.
      – plug in ethernet cable to router
      – get an ip address
    6. Observe the results of the implementation.
      – during this process we got stuck as another problem was encountered. Now we need to decide to investigate and resolve this issue or revert back to previous build and not resolve these two issues directly
    7. Repeat the process if the plan does not resolve the problem.
      – now we have a new issue to work on and can follow this exact same process to see if we can fix the ethernet issue
    8. Document the changes made to solve the problem. – This should be step one. Start by copying this block of text and editing it for your specific use case…
      – open a support case
      – capture notes as you go along to have accurate records about exactly what steps were taken
      – since we’re using an insider build we agreed to share feedback about what issues we encountered and help make the next version better for others

    Troubleshooting Ethernet on Acer Laptop after Windows 10 upgrade

    1. Define the problem.
      – ethernet network wouldn’t get an IP address after the OS upgrade
    2. Gather detailed information.
      – Windows 10 Insider Build 21292 rs prerelease 210108-1514
      – Laptop is Acer Aspire A715-71G
    3. Consider probable cause for the failure.
      – drivers need to be updated for new OS install
    4. Devise a plan to solve the problem.
      – download new drivers from Acer: https://www.acer.com/ac/en/US/content/support-product/7296?b=1&pn=NX.GP8SV.005
      – since our wireless networking isn’t working and now we know the Ethernet isn’t working too we need to use another computer to download the drivers to see if that resolves this issue
      – once the drivers are downloaded on another computer we can copy them to a USB Flash Drive (UFD) and then use that UFD to move them to the device having the network issues
      – install new drivers
    5. Implement the plan.
      – update drivers
    6. Observe the results of the implementation.
      – get an ip address
    7. Repeat the process if the plan does not resolve the problem.
      – now that we solved this issue we can follow this same process to see if we can fix the wireless issue
    8. Document the changes made to solve the problem. – This should be step one. Start by copying this block of text and editing it for your specific use case…
      – capture notes as you go along to have accurate records about exactly what steps were taken
      – since we’re using an insider build we agreed to share feedback about what issues we encountered and help make the next version better for others
      – to prepare for future update problems with drivers it is suggested that we keep a copy of all drivers needed on the local hard drive or on a USB drive handy for this type of situation

    MITRE ATT@CK for the Global Security Operations Center GSOC CyberSec

    The ATT&CK framework from MITRE is focused on techniques used to compromise client operating systems such as Microsoft Windows, Linux, Apple’s Mac OS, and mobile os like Apple iOS and Google Android.

    Adversarial
    Tactics,
    Techniques,
    &
    Common
    Knowledge

    But as we’ve seen recently lateral attack from one of these client OS devices can be used against servers and cloud resources too such as stealing an OAUTH token allowing admin access for the SAML SSO solution and gaining access to pretty much any SaaS tool used at the organization.

    MITRE allows external contributors but this process needs to be enhanced to more easily allow vendors and subject matter experts to update content and provide feedback.

    If you work in or are building a SOC then this is for you. MITRE has a book published in 2014 by Carson Zimmerman. Download the PDF file here: Ten Strategies of a World-Class Cybersecurity Operations Center

    Table of Contents
    Executive Summary