PRODUCT INFORMATION
Abnormal
4Data partners with Abnormal to provide comprehensive email protection against attacks that exploit human behaviour, including phishing, social engineering and account takeovers, with a platform that deeply understands human behaviour.
Protect your organisation from socially-engineered email attacks that target human behaviour.
PRODUCT INFORMATION
CrowdStrike
CrowdStrike offers the world’s most advanced cloud-native platform that protects and enables the people, processes and technologies that drive modern enterprises. The CrowdStrike Falcon® platform is a powerful addition to 4Data’s portfolio and covers the complete defence-in-depth spectrum – from endpoint security to identity management, SIEM and cloud security.
Total protection has never been easier.
Take advantage of our CrowdStrike FREE 15-day trial:
PRODUCT INFORMATION
Centripetal
Centripetal, a global leader in intelligence powered cybersecurity, is operationalising the world’s largest collection of threat intelligence, in real-time, to protect organisations from every known cyberthreat through its innovative patented technologies. CLEANInternet® from Centripetal provides the fastest packet filtering patented technology on the market to deliver robust security at scale.
PRODUCT INFORMATION
Splunk
Find out more about our unique approach to Splunk consultancy
Read our top tips of what to look for when choosing a Splunk Professional Services Partner
See what Splunk Professional Services we offer
HOW TO GUIDES
Long-term Storage for GoBD
The whole setup can also be used to simply store your important data in an archive outside your data platform (in our case, Splunk Enterprise).
This can be beneficial in multiple ways; to fulfil legal requirements; to reduce the cost of storing the data on fast searchable storage, or simply to create offsite storage.
The technologies we will use here are:
Storage: AWS S3 (Glacier) Deep Archive
Routing: Cribl Stream
Audit/Searching: Splunk Enterprise
In this scenario, everything is already located in AWS. If you are using an on-prem environment, you will need to have secure upstream to AWS S3 for this setup to work securely.
If you have specific requirements that you would like to discuss with us here at 4Data, simply get in touch via the contact page or the Chatbot.
Pre-requisites (all located in AWS):
To store something in AWS S3, we need an S3 Bucket. To create a bucket we simply log in to the AWS Console and switch to the S3 UI. Here we create a new bucket.
In our case, this will be a Bucket named “gobd_demo”.
Choose the region in which your Cribl Stream instance lives.
Now we will change some settings to fulfil some of the requirements
for GoBD compliance.
1. Disable Public Access
2. To add encryption at rest we will have to enable encryption:
3. Enable Object Lock
The bucket can now be created, after which we will make some further adjustments to the bucket properties.
4. (Optional) If you track/index your AWS data in Splunk for additional auditing capabilities, then enable server access logging.
5. Once the bucket is created there are some other steps to complete for compliance with the GoBD regulation: Edit the object lock to enable retention and set the mode to “Compliance”. This will keep the data for the allotted retention period and no one can override or delete any of the objects during that period.
6. Adjusting the permissions on the bucket policy.
This is what the policy should look like:
7. The Bucket ACL (Access Control List) should be adjusted to look like this so that only entities in your AWS Account can read/write to that bucket.
8. Now the last bit is to give your Cribl Stream instance access to that bucket.
To achieve this, we will attach the following policy to the instance role.
You can achieve this by simply moving to IAM and creating a new policy. Paste the example shown here. After the policy has been created, move to the IAM Roles and select the role used on your Cribl Stream instance. Simply attach the policy you just created.
Now that we have set up AWS S3, as well as the policies for access to the bucket, let’s start setting up Cribl Stream.
To set up Cribl Stream to send to both S3 and Splunk, we need to configure them as destinations. In Cribl Stream, navigate to Destinations and select Splunk Single Instance, or if you are using a distributed environment, Splunk Load Balanced. Create a new destination with the details of your Splunk instance and ensure the appropriate ports are open in your AWS security policy. Do the same for the S3 bucket, under Destinations, Amazon S3.
To grant Cribl Stream access to the S3 bucket, adjust the settings of the bucket destination as follows:
1. Assuming Cribl Stream is running in the same environment as your S3 bucket, select the ‘Auto’ authentication method.
2. Under ‘Assume Role’, enter the ARN of a role with sufficient access to your S3 bucket. This role should be able to read, write and restore objects in your S3 bucket.
Back under General Settings, there are some options available to you to format the structure of the bucket when Cribl Stream stores your data.
1. The ‘Key Prefix’ determines the name of the root directory created
in the S3 bucket
2. The ‘Partitioning Expression’ allows the use of a JS expression
to determine how the files are partitioned, which can include
fields from the events themselves
Our example stores all files under a root directory named ‘gobd’ and stores files in subsequent directories ordered by year/month/day/index
Navigate to Advanced Settings and select the following:
1. Under ‘Object ACL’ select ‘Bucket Owner Full Control’
2. Now, to satisfy the long-term storage requirements, select the storage class that you would like Cribl Stream to store the data in
Now, Cribl Stream is able to send events to both our S3 bucket and our Splunk environment. We also need to configure the collector which will pull the restored events from S3 in the event of an audit.
Navigate to Data -> Sources and filter by Collectors. Select the S3 collector and create a new collector. Ours will be named ‘gobd_collector’. After naming the collector, select the S3 destination we just configured in the ‘Auto-populate from’ drop-down to automatically configure the collector with the correct settings from our bucket.
Our collector is now set up. The collector also provides a ‘Path’ variable similar to the S3 destination. This determines where the collector looks for events and allows the folder values to be tokenised into fields within the event. This is where we create the “S3_*” fields used in our pack.
Now we are ready to configure our Routes and Pipelines to send the data to the relevant destinations. Fortunately, we have created a pack containing the majority of the config required here to make this easier. Download it here and install it via the Cribl Stream UI.
Once installed, open the pack and you will see two routes configured.
The first takes any data sent to it that doesn’t come from our collector and processes it ready to be sent to S3. The second takes any data that does come from our collector and processes it ready to be sent to Splunk. If you configured your collectors with different names, make sure to update the route filters here.
Now we just need to associate these routes with their destinations. To do this, create two routes outside the pack:
1. Create a filter that selects the GoBD data you want to archive in S3, then select the GoBD_Restore pack as the pipeline and the S3 bucket as the destination.
2. For the second route, use the filter from the Pack which selects events from our GoBD collector. Set the pipeline to the GoBD_Restore pack and set the destination to Splunk.
We want to have a very simple audit process so the GoBD auditor doesn’t necessarily need SPL knowledge. To achieve this, we will utilise our 4Data-made GoBD Audit App. This will cover some very important pieces of compliance.
1. Being able to set up the Audit Dashboard with different fields of interest
2. The audit ability itself
3. Ability to track access to the GoBD bucket
4. Alert on changes to the bucket (This shouldn’t be possible with the settings we have used to set up the bucket)
5. Log the audits done
6. Audit who has accessed the GoBD audit index (this might be added depending on the progress of development)
7. Delete audit data in Splunk when finishing the audit
You can create your own dashboards to achieve the same results.
We utilise a couple of searches to achieve those pieces of compliance.
These are the Dashboards in our App:
GoBD Bucket Access Tracker
GoBD Index Access Audit
Who has accessed the GoBD index and when?
GoBD Audit Dashboard
This dashboard will walk the auditor in a “Wizard” style through the selection of fields and values he would like to check. This is highly customizable.
Here you can select the number of days you would like the restored data to be available for and the speed of the retrieval. The speed will depend on which storage class you chose to use to store the data when configuring the S3 destination in Cribl.
Once restored, you will see a banner at the top of the page on the event informing you of the restore status and the time remaining for the availability of the data.
At this point, the data is ready for Cribl to collect and re-ingest into Splunk.
In Cribl, navigate to Data, Sources and filter by collectors. Select the S3 collector and on our GoBD collector select Run.
This will show the collector screen where we can retrieve our data. Write a filter to select the data you would like to retrieve. This can include the fields created in the Path variable from the collector configuration created earlier.
Running the collector in Preview mode will show a sample of the events that will be returned when the collector is run. Running in Discovery mode will display the total number of events that will be returned by the collector, and running in Full Run mode will do what we need and return the events themselves.
Select Full Run and run the collector.
And that’s it! Cribl will automatically route the collected data to our GoBD pack and onto the configured Splunk index. It’s time to jump into Splunk and take a look at the data.
Conclusion
As we have seen, by combining data routing solutions like Cribl Stream, AWS S3 and Splunk Enterprise, it is possible to achieve what other provides of proprietary solutions get paid a pretty penny for.
This solution can also be implemented to store data for a long time in cheaper storage, with the flexibility to restore and search through this at any point later.
In today’s data-driven world, ensuring secure and immutable storage while complying with GoBD regulations is a challenge. However, a comprehensive solution that uses AWS S3, Cribl Stream and Splunk Enterprise can make this process easy and cost-effective. This solution offers long-term storage, data duplication and auditing capabilities, all of which are essential for organisations that need to adhere to GoBD guidelines.
By following the step-by-step instructions provided in this guide, organisations can set up the solution quickly and efficiently, while also ensuring the security and integrity of their data. With the ability to restore and search through data at any point, this solution is not only compliant, but also flexible and scalable, making it an ideal choice for businesses of all sizes.
So, if you’re looking for a secure and compliant storage solution for your organisation, look no further than AWS S3, Cribl Stream and Splunk Enterprise. If you like the solution, but are unsure whether this can be implemented by you or your team, 4Data Solutions’ Professional Services team is here to help.
