Democratising Data Analytics

Democratising Data Analytics

Over the past 20 years, data analytics has grown from its infancy as a tool for sysadmins to a platform for practically any use case. Along the way, we’ve seen incredible innovation and progress in how these platforms have developed. We’ve also witnessed the exponential growth in data volumes and the, initially three, then four, and then five ‘Vs’ of big data (velocity, volume, value, variety and veracity).

But we have now reached a point where data analytics has become prohibitively expensive for many. When Tier 1 banks have to cautiously deliberate what data they really need to analyse, you know that things have become just a little too costly.

Time then for a new, more affordable way to do data analytics.

But how? We’ve been here before with promises of amazing new technologies. The cloud promised boundless scalability at an affordable price, but many have been bitten by escalating costs. But that doesn’t mean the cloud is the wrong choice; it was just the wrong choice for traditional data analytics platforms that were never designed to run in the cloud.

Enter the first, truly cloud-native data analytics platform, Axiom.

So, what makes Axiom’s approach different and affordable?

There are three important design decisions that make Axiom an affordable data analytics solution for the masses:

1. Storage

    For some time now, storage has been an expensive part of the infrastructure necessary to do data analytics, and whilst we enjoyed the leap in speed that SSD brought us, we didn’t enjoy the price tag that went along with it. Object storage, like AWS S3 (simple storage service), is very inexpensive but in the past didn’t lend itself to being used for analysis and was instead used as a storage medium for large volumes of data that only required infrequent access. Axiom figured out a way to use object storage for all of its requirements, which has led to significant cost savings. The key to making this work is the file system, which I’ll discuss below.

    2. Compute
    Typically, an army of compute instances has to run perpetually to service the data ingest and the users’ searches. This is expensive and goes against the cloud philosophy of embracing the fluctuations of dynamic workloads and switching off or turning down resources when demands allow, thus saving money and making cloud computing affordable. Axiom based its analysis engine on serverless functions, doing away with the costly compute infrastructure altogether. Again, this has resulted in a significant cost reduction.

    3. File system
    Finally, to make this all work, Axiom needed to figure out how to make object storage useful for search. To do this, they tried out some open-source file systems like Parquet but quickly concluded that nothing existed which could provide the speed and scale required for modern data analytics. So they wrote their own columnar file system format. This allows Axiom’s customers to leverage low-cost object storage for all their needs, whilst maintaining fast search response times.

In conclusion, Axiom has created an affordable data analytics solution for the masses and in doing so, has democratised data analytics.

Data onboarding is a breeze, and their variable schema means that you don’t need to worry about the format too much; just send your data to Axiom and you can use it immediately.

Today, Axiom is offering to store forever at no extra cost, and whilst I’m sceptical that this can scale indefinitely, the point is that retention is no longer an issue. Need to keep data for one year or ten years; no problem!

Get in touch to find out more about the first truly cloud-native data analytics platform. Call us on +44 330 128 9180 or email info@4datasolutions.com.

Data analytics: Will your business survive the ‘perfect storm’ unscathed?

Data analytics: Will your business survive the ‘perfect storm’ unscathed?

If this headline caught your attention, it was meant to. Businesses are facing a ‘perfect storm’ of challenges – we are all operating in a ‘VUCA’ world, where volatility, uncertainty, complexity and ambiguity all serve to make running a business more challenging than ever before.


In this, the first of a two-part blog series, we explore the factors behind this perfect storm, the pressing need for businesses to glean valuable data insights, and the challenges associated with implementing and using a data analytics tool, using Splunk as an example.

Although the global economy is showing signs of gradual recovery from the pandemic and Russia’s invasion of Ukraine, many challenges are still clouding the horizon. Economic growth remains weak, global activity is losing momentum, the threat of inflation is ever present, and businesses are still facing increasing costs.

Against this uncertain economic backdrop, data continues to explode. An ever-increasing volume of unstructured data needs to be stored, managed and analysed. According to Cybersecurity Ventures, the total global data storage will exceed 200 zettabytes of data by 2025.

Leveraging value from data and keeping it safe have rapidly become board-level issues. Businesses are adopting a data-driven culture to avoid falling behind their competitors. Directors are finding themselves in the precarious situation of needing to simultaneously keep their costs down and sustain growth, all while managing increasing amounts of data.

The need for data analytics platforms has never been greater as a means of gleaning valuable data insights for new operational and business efficiencies. In a recent survey by software intelligence company Dynatrace, 59% of CIOs said that without a more automated approach to IT operations, their teams could soon become overloaded by the increasing complexity of their technology stack. But with the data explosion showing no signs of slowing down and the global economy still not out of the woods, there is a very real risk that the costs of managing data analysis could explode in the same way that data volume is exploding.

Analytics platforms are expensive, with businesses facing not just the considerable upfront investment but also the often sky-high ongoing licence and storage costs. Businesses need to optimise their data analytics platform spend while at the same time manage business risk and identify what to prioritise for observability.

In the context of the factors we’ve explored above, let’s look at the leading data analytics platform, Splunk, and consider the challenges associated with its implementation and usage.

Splunk is widely recognised as a powerful and highly effective data analytics platform for Security, Service Management and Observability. It is a tool that makes it very quick and easy to get value from. However, to properly architect an Enterprise Splunk deployment takes far more effort, and preferably, outside help from certified consultants. Splunk Cloud takes away this architectural requirement but there are still topics that need to be addressed that ideally require skilled help. For example: data curation, naming conventions, knowledge artefact management, permission management, integration with a wide selection of 3rd party tools, search optimization, advanced dashboard and reporting, etc. Many businesses that use Splunk may not have the inhouse skills or knowledge to fully exploit its features and functionalities in line with specific requirements, or to fully integrate it with other systems within the business. Many businesses fail to gain return on their Splunk investment and miss out on the totality of the opportunity Splunk presents.

Today’s uncertain economy and associated business pressures, the data explosion and cost of data analysis has altogether created a perfect storm of challenges. Never has it been more important to ensure your chosen data analytics platform is adding defined business value and supporting business drivers and outcomes. If you are a user of Splunk or considering investing in it – identifying a highly experienced Splunk Professional Services Partner is critical. Look for a partner experienced in your sector and size of business, with granular knowledge of Splunk’s features and functionality and a team of consultants fully qualified and capable of working with you to align your Splunk implementation directly with your specific business needs.

In part two, we will explore this further and provide tips on what to look for in a strong Splunk partner and the difference it makes to work with consultants that can really make Splunk WORK for you, whether for security, systems and infrastructure purposes or for its enormous value as a business intelligence tool for CEOs.

Further reading

What to look for in a Splunk Services Partner – https://4datasolutions.com/services/splunk-services/the-ideal-splunk-partner/