What is our primary use case?
Typically, the standard approach for Splunk sizing involves gathering data from the entire IT environment, regardless of whether it's hardware, virtualized, or application-based. This data is then collected and monitored through Splunk as a comprehensive security solution. We also work with Splunk-related platforms like Application Performance Monitoring to provide a holistic view of system performance. Recently, we implemented this solution for a bank in Jetar. Splunk excels at collecting high-volume data from networks, making it ideal for performance monitoring and scaling. During the sizing process, it's crucial to calculate the daily data ingestion rate, which determines the amount of data Splunk Enterprise needs to process and visualize for security purposes. Several factors need consideration when sizing Splunk: tier structure hot and cold buckets, customer use cases for free quota access, and storage choices based on data access frequency. Hot buckets typically utilize all-flash storage for optimal performance and low latency, while less frequently accessed data resides in cold or frozen buckets for archival purposes. In essence, the goal is to tailor the Splunk solution to meet the specific needs and usage patterns of each customer.
One challenge that our customers face is slow data retrieval. Customers may experience delays in retrieving call data due to complex search queries within Splunk Enterprise Security. These queries can sometimes take up to an hour and a half to execute. Our architecture incorporates optimized query strategies and customization options to significantly reduce data retrieval times. This enables faster access to both hot and cold data.
Another challenge is scalability constraints. Traditional solutions may have limitations in scaling to accommodate increasing data volumes. This can be a significant concern for customers who anticipate future growth. Our certified architecture is designed for easy and flexible scalability. It allows customers to seamlessly scale their infrastructure based on their evolving needs, without encountering the limitations often faced with other vendors' solutions.
The final challenge is complex sizing and management. Traditional solutions often require extensive hardware configuration and sizing expertise, which can be a challenge for many organizations. This reliance on hardware expertise can hinder scalability and adaptability. Our architecture focuses on software and application administration, minimizing the dependence on specific hardware configurations. This simplifies deployment and ongoing management, making it more accessible to organizations with varying levels of technical expertise.
Our architecture leverages Splunk's native deployment features, including:
Index and bucket configuration. Data is categorized into hot, warm, and cold buckets for efficient storage and retrieval. Active/passive or active/active clustering. This ensures high availability and redundancy for critical data. Resource allocation. Data, compute, and memory resources are distributed evenly across clusters for optimal performance.
For high-volume data ingestion exceeding 8 terabytes per day, we recommend deploying critical components on dedicated physical hardware rather than virtual machines. Virtualization can introduce overhead and latency, potentially impacting performance. Utilizing physical hardware for these components can help mitigate these bottlenecks and ensure optimal performance for large data volumes.
How has it helped my organization?
Splunk Enterprise Security provides visibility across multiple environments. IT leaders and management directors often seek a simplified monitoring tool that can handle everything. However, using a third-party tool or a monitoring tool for multiple environments comes with certain considerations. These may include software version upgrades, connector updates, or API integrations for collecting specific metrics beyond the usual ten. Therefore, the key factors for a customer choosing a monitoring solution are, how easily can the tool integrate with existing physical, virtual, microservices, or hyper-scaler environments, whether it can provide a centralized view of monitoring data across multiple environments, and whether it can integrate with existing data analytics tools like Cloudera, Starburst, or Teradata. Integrating a monitoring solution with data analytics is crucial for a complete picture. While a standalone monitoring solution can help with capacity planning, data analytics provides insights for code analysis and historical data. This allows management to plan budgets, reduce costs, and make informed decisions for the future. Combining a monitoring tool like Splunk Enterprise Security with a data analytics engine like Cloudera or Teradata maximizes the value of data and empowers better decision-making.
Our monitoring tools offer various functionalities, including detection and third-party integration. For example, we have an integration with TigerGuard, a platform for threat detection. Additionally, we provide robust auditing capabilities to track changes within the environment. This helps identify potential intrusions and suspicious activity, whether from internal or external actors. To ensure the security of our monitoring tools, we implement several prevention and protection mechanisms. This includes continuous monitoring of logs and audits, even in case of tool failure. Leading enterprise monitoring solutions often connect to dedicated audit servers via SNMP traps, providing a centralized view of all infrastructure changes. This allows administrators, like Splunk users, to easily track modifications and identify potential security risks. Furthermore, individual software products within our monitoring suite have their access control lists and security measures. These may include features like certificates, user authentication, and security manager integration. Additionally, some products offer optional plugins or add-on licenses to enhance their auditing capabilities and meet specific organizational security requirements. Security is a complex and multifaceted topic, encompassing various aspects. This includes data location, user activity monitoring, intrusion prevention, and incident recovery procedures. Addressing these concerns effectively requires a comprehensive security platform assessment that evaluates the entire system, from hardware to applications, ensuring data integrity, encryption, and overall security at every layer.
Threat intelligence management utilizes dedicated tools for both threat and incident management. These tools help organizations define their response plan in case of an event, including how to recover, what the RTO and RPO are, and how to achieve them. This ensures the organization can recover quickly and efficiently in the event of a failure, unauthorized access, or data deletion. While threat incident management strategies may vary depending on the customer, the banking sector typically undergoes rigorous threat management inspections. While I may not be a threat management expert, there are crucial security measures to consider, encompassing personnel training, hardware security, and application controls. These elements, when orchestrated harmoniously, contribute to a secure environment that minimizes the risk of breaches, facilitates successful audits, and ensures data integrity.
The effectiveness of the threat intelligence management feature depends on how the customer responds to various threats, such as ransomware or network intrusions. While the tool provides recommendations, it requires customization to align with each organization's unique categorization criteria like high, medium, low, and specific security objectives. Ultimately, the goal is to protect data, enhance security, and ensure effective incident response procedures. Deploying the threat intelligence tool necessitates customization for each customer. Default settings may not be optimal, as human intervention might be necessary to address potential software errors or inaccurate recommendations. In such cases, manual intervention might be more effective. Therefore, the tool's usefulness depends on the specific threat, its recommendations, and the organization's response approach.
Splunk Enterprise Security is a powerful tool for analyzing malicious activity and detecting breaches. However, its effectiveness depends heavily on proper configuration and skilled administration. They must be able to connect Splunk with the necessary parameters, collect logs daily, and analyze them effectively. They must also have the ability to query Splunk efficiently to gather relevant data, an understanding of use cases and how to integrate with other systems securely, customization of the environment to meet specific needs, including adding connectors and add-ons, and visualization of data in a way that is clear and actionable for both analysts and management. While Splunk is a valuable platform, it requires careful management and expertise to unlock its full potential. Companies deploying Splunk should invest in skilled administrators to ensure its effectiveness in securing their environment.
Splunk helps us detect threats faster. As a Splunk administrator, I can monitor for suspicious activity, such as sudden changes in behavior, high resource utilization on specific file shares, or unusual data transfers. These events can trigger questions, like, Why is the system experiencing high utilization, is data leaking and being transferred elsewhere, why is this application consuming excessive resources, why has data suddenly disappeared from the system? Splunk Enterprise Security provides valuable insights and helps identify potential security issues. However, integrating threat intelligence management with Splunk can further automate this process. When suspicious activity is detected, the system can automatically take predefined actions. However, these actions require customization and testing before implementation. This may involve, customer review and approval of the automated response, POC testing to validate the effectiveness of the response, regular monitoring of the system's behavior and response to threat intelligence, and fine-tuning or customization of Splunk Enterprise Security settings to optimize threat detection and response.
Splunk has been beneficial to our organization from a partnership perspective. Even after Dell's acquisition of Cisco, our strong collaboration and certified solutions continue. This partnership strengthens our position with customers seeking the best solutions on the right platform. For example, if a customer requires a Splunk solution and a competitor lacks certified solutions, it could hinder their trust and purchasing decision. In contrast, our close collaboration with Splunk and certified add-ons for Splunk Enterprise Security adds value. We possess expertise in various Splunk architectures. I've worked with over four banks in Saudi Arabia alone that utilize Splunk and Dell hardware. Globally, we cater to diverse Splunk architectures and platforms, ensuring customer satisfaction with our diverse technology expertise. While we acknowledge competition, the focus here is on how our partnership with Splunk enhances the integration experience, offering both tightly coupled and loosely coupled architectures.
Since implementing Splunk Enterprise Security, we've observed improvements in both stability and the accuracy of data visualization. The low latency allows us to efficiently query the extensive data it provides. Splunk goes beyond collecting basic metrics like CPU or memory utilization; it comprehensively gathers data from various sources, including networks, applications, and virtualization. This unified platform eliminates the need for siloed solutions and enhances the capabilities of existing engineering software. While Splunk is a popular choice for data analysis due to its powerful features, its pricing structure based on daily data ingestion can be expensive. This pricing model, however, allows them to accurately charge based on resource usage. It's important to consider your data collection and visualization needs to determine the appropriate licensing tier. While other monitoring tools might share similar pricing models, Splunk distinguishes itself through its data segregation across various components. This simplifies communication between indexes, forwarders, and searches, allowing for efficient data processing within a single platform. Additionally, Splunk excels in data visualization and analytics, making it a leading choice for security and observability solutions. Their recent top ranking in Gartner's observability category further emphasizes their strengths. This recognition stems from their platform's compatibility with diverse hardware vendors, exceptional data visualization capabilities, and innovative data segregation strategies. Splunk's tiered access control and efficient cold/frozen data storage further enhance its value proposition. Ultimately, Splunk empowers users to interact with their data effectively. This valuable asset, when properly understood and visualized, can provide actionable insights without impacting network or application performance. Moreover, Splunk's customization and implementation potential extend beyond data analysis, offering recommendations and threat intelligence for proactive security measures. In conclusion, while Splunk's pricing might initially appear expensive, its comprehensive features and capabilities justify its cost for organizations seeking advanced data analysis and security solutions.
Realizing the full benefits of Splunk Enterprise Security takes time. While the software itself can be deployed quickly, it requires historical data to function effectively. This means collecting data for some time before you can rely on it for accurate insights. Several factors contribute to the time it takes to see value. First, there is deployment and customization. Setting up Splunk involves hardware, software, and integration, which can be time-consuming, especially during the first year. The second is data collection. Building a historical data set takes time, and the initial period may not provide significant value. There is also customization and training. Tailoring reports and training users requires additional investment, potentially involving workshops and professional services. To expedite the process, Splunk offers various resources including, proof of concept which allows testing Splunk with a limited data set for a specific period. Splunk may offer temporary free licenses for small workloads to facilitate initial evaluation. Splunk provides educational resources to help customers understand and utilize the platform effectively. Additionally, some partners leverage their Splunk expertise to help customers. Partners can educate and guide customers through the process, streamlining their experience, and assist with customizing reports and training users, accelerating the value realization process. By understanding these factors and leveraging available resources, organizations can optimize their Splunk implementation and achieve its full potential within a reasonable timeframe.
What is most valuable?
Splunk has been recognized by Gartner as a leader in providing visibility for observability and monitoring across various platforms, including physical, virtual, and container environments, for several years. This has made it a popular choice for many organizations, including those in the banking industry. Currently, only one of our banks utilizes QRadar. This may be due to the cost associated with switching from Splunk, which can be expensive. As a result, the customer might be prioritizing financial considerations over functionality at this time. It's important to note that while Splunk is recognized as a leader in platform capabilities, the decision to use a specific solution should ultimately be based on both functionality and cost considerations. This is why we have established a joint engineering team with Splunk to develop a platform that meets the needs of our customers.
What needs improvement?
I'd like a dashboard that allows me to connect elements through drag-and-drop functionality. Additionally, I want the ability to view the automatically generated queries behind the scenes, including recommendations for optimization. This is just a preliminary idea, but I envision the possibility of using intelligent software to further customize my queries. For example, imagine I could train my queries to be more specific through an AI-powered interface. This would allow me to perform complex searches efficiently. For instance, an initial search might take an hour and a half, but by refining the parameters through drag-and-drop and AI suggestions, I could achieve the same result in just five minutes. Overall, I'm interested in exploring ways to customize queries for faster and more efficient data retrieval. Ideally, the dashboard would provide additional guidance and suggestions to further enhance my workflow through customization and optimization.
For how long have I used the solution?
I have been using Splunk APM for four years.
What do I think about the stability of the solution?
The stability of Splunk Enterprise Security depends on how data is tiered. Splunk recommends different storage options based on data access frequency and volume. Hot and warm data: This data is accessed frequently and requires fast storage like SSD or NVMe. Cold and frozen data: This data is accessed less often and can be stored on cheaper options like Nearline, SaaS, NAS, or object storage.
Splunk prioritizes cost-effectiveness and recommends low-tier storage for cold and frozen data, which typically makes up the majority of customer data. This reduces costs compared to expensive SAN storage. However, the decision ultimately depends on the customer's budget and specific needs. Customers with limited budgets or small use cases might choose to store all data on a single platform initially and expand to dedicated cold and frozen storage later. This approach requires manual configuration changes e.g., modifying index.conf and forwarder configurations to redirect data to the new tier.
While Splunk recommends optimal tier configurations for hot, warm, cold, and frozen data, the final decision rests with the customer based on their budget and specific requirements.
Changes to storage tiers can be implemented later through configuration adjustments, but this process might be more complex than using dedicated storage from the beginning.
Overall, Splunk guides the best tier options for different data access patterns while acknowledging the customer's autonomy in making the final storage decision.
What do I think about the scalability of the solution?
We have ambitious expansion plans. As our customer base grows, we see significant increases in data ingestion. For example, one of our largest Splunk customers has increased its daily data ingestion from two terabytes to eight terabytes in just three years. This expansion benefits both the customer and Splunk. The customer gains valuable insights from the additional data, while Splunk increases its revenue through additional license sales. However, it's important to note that expanding data ingestion requires careful consideration of hardware limitations. Increasing data volume necessitates adding more forwarders, indexes, and searches, which can impact Splunk licensing requirements. This highlights the crucial need for comprehensive planning and resource allocation during expansion initiatives. Furthermore, we have observed instances where customers unintentionally exceed their licensed data ingestion capacity. For example, one bank was ingesting six terabytes of data per day while only holding a four-terabyte license. This underscores the importance of close monitoring and proactive license management to ensure compliance and avoid potential licensing issues.
Which solution did I use previously and why did I switch?
Our expertise extends beyond Splunk. We offer certified architectures for other SIEM solutions like IBM QRadar, catering to diverse customer requirements. However, IBM QRadar does not have as wide of a platform as Splunk Enterprise Security.
What's my experience with pricing, setup cost, and licensing?
Splunk can be expensive, as its licensing is based on the daily data ingestion volume. While we've observed numerous implementations, most are executed remotely by Splunk itself. However, if on-site assistance from a Splunk engineer is desired, it can be costly due to travel expenses from either the Dubai office or Europe. To address this, Splunk is exploring partnerships to offer implementation services at more accessible price points.
For someone evaluating SIEM solutions and prioritizing cost, traditional marketing materials may not be the most effective approach. Customers in Saudi Arabia, like many others, often appreciate tangible demonstrations of value. Therefore, consider offering a POC to showcase Splunk's capabilities in their specific environment. Investing in the customer through various strategies can demonstrate your commitment and build trust. Granting temporary access allows them to experiment with Splunk firsthand. Provide resources and support to help them learn and utilize the platform effectively. Leverage your partner network to offer additional training and expertise. Invite key decision-makers to exclusive events or meetings with Splunk leadership, fostering a deeper connection and understanding. Remember, success often hinges on addressing specific needs. While POCs and business cases are crucial, consider potential customization requirements and existing workflows. If they've used a different tool for years, transitioning may require additional support and training due to established user familiarity. Splunk's investment in the customer journey goes beyond initial acquisition. By offering POCs, temporary licenses, training, and even exclusive experiences, you demonstrate value and commitment, ultimately fostering long-term success. Demand generation, in essence, boils down to two key aspects, Identifying their specific requirements and desired outcomes, and recognizing that different customers have varying budgets, experience levels, learning curves, expectations, and decision-making processes. While some customers may be more challenging to persuade, others readily embrace the extra mile. Enterprise clients often fall into the latter category due to their greater flexibility in resource allocation, dedicated security operations teams, and ability to invest in necessary hardware. Remember, SIEM solutions often involve hardware considerations beyond just software, so understanding these additional costs is crucial for accurate solution sizing and customer budgeting.
Which other solutions did I evaluate?
Several competitors to Splunk exist in the market, including IBM QRadar, AppDynamics which is used by some customers for monitoring and security, and Micro Focus used for enterprise monitoring, incident reporting, and capacity planning. While Dynatrace is a leader in the field, its presence in the banking sector, particularly in Saudi Arabia, seems limited, perhaps due to having only one certified partner acting as its distributor. In contrast, Splunk boasts a wider network of partners who actively implement and enable customers, leading to its increased market prevalence.
What other advice do I have?
I would rate Splunk APM a nine out of ten.
Monitoring multiple hyperscalers with a single tool can be challenging. While some tools like VMware CloudHealth offer limited cross-platform capabilities, they often focus on specific aspects like virtual instances and storage. For comprehensive cloud monitoring across different hyperscalers like Azure and AWS, third-party solutions are typically necessary. Here at Dell, for example, we focus on monitoring tools for our own workloads and installed base, allowing integration with third-party solutions for cloud environments. This enables customers with workloads across multiple hyperscalers to leverage established enterprise monitoring tools like New Relic, AppDynamics (Cisco), Micro Focus (HP), and Splunk for unified visibility. Ultimately, choosing a solution often involves balancing operational and capital expenditures. By employing third-party tools, organizations can achieve comprehensive monitoring across various cloud environments while potentially reducing overall costs.
We offer various deployment options for Splunk to cater to diverse customer needs and regulations. We can deploy Splunk on various infrastructures, including hyper-converged, bare-metal, two-tier, and three-tier architectures. While cloud deployment is an option, regulations from the Saudi Central Bank restrict customer data storage outside the kingdom. Therefore, most of our customers in the financial sector opt for private or local cloud solutions. While a dedicated private cloud experience for Splunk isn't currently available, customers are seeking access to features like the SmartStore, a caching tier that is now bundled with the Enterprise Security license previously offered separately from version 7X onwards. The chosen deployment approach depends on factors like budget, customer expectations, performance requirements, and compatibility with Splunk's recommended sizing solutions. We utilize both internal sizing tools and Splunk's official tools to ensure proper resource allocation for indexers, search heads, and forwarders based on specific customer needs. We have deployed our Dell servers, storage, and data protection solutions. Additionally, we have implemented a reference architecture. From a hardware perspective, we have everything in place to support Splunk as a reference architecture. This is indisputable, as it reflects our current infrastructure.
I have one customer who uses Splunk on a single site. In contrast, other customers have deployed Splunk in an active-active cluster configuration across two sites, effectively segregating the data across the environments with two-factor authentication. For these other environments, I have observed that each customer has a unique monitoring perspective or performance requirement, reflected in their individual subscriptions.
Splunk is responsible for software maintenance, while we handle the hardware aspects.
Splunk Enterprise Security is one of the most mature security solutions available. While it is expensive, it offers good value by providing the necessary security measurements, monitoring, and auditing capabilities required for running an enterprise environment.
The combined forces of Splunk and Dell create significant resilience for us. Our joint architecture, strong alignment between the Dell account team and Splunk sales and presales, and collaborative efforts have been instrumental in addressing specific customer needs, such as sizing. This collaboration is mutually beneficial: Splunk focuses on selling licenses, while Dell prioritizes hardware sales. Unlike Cloudera, which optimizes licenses for its platform, Splunk bases licensing on the ingestion rate, demonstrating its alignment with our advanced architecture. This creates a win-win situation for both companies.
Which deployment model are you using for this solution?
On-premises
Disclosure: My company does not have a business relationship with this vendor other than being a customer.