Is it possible?
I have been building ELK stacks for almost 4 years and have come to the conclusion that cloud based ELK stacks for complex BI can be built within $50/month budget. Any lower than this could compromise the stability of the stack. With the below mentioned AWS reserved instances, we can reduce the cost to under $30/month
I started out by creating a simple ELK Stack on 2x cloud servers and installing the connecting ElasticSearch first. I encourage you to use the following sequence when installing & configuring the ELK.
- ElasticSearch 6.8 or 7.x (Or any other version)
- Cerebro for ElasticSearch (Install this to manage the index)
- Kibana (The main visualizer)
- Excelastic (To import xsl or csv files into ElasticSearch)
- Curator (Automate index management by creating rules)
- Beats (Transfer log files from source to LogStash -over TCP)
- APM (Application Performance Monitoring)
There are many many documents out there with installation instructions for the ELK stack [ https://community.spiceworks.com/how_to/164166-installing-elk-7-elasticsearch-logstash-and-kibana-windows-server-2016-part-1]
ELK Stack For $150
A moderate size ELK stack would cost between $100 to $200 based on the capacity requirements. I have selected a basic $150 Master-Slave architecture with ElasticSearch query processor. This is a special ElasticSearch node with Data=False & Master=False configuration. Such nodes would automatically participate in processing log files.
ELK Stack for above $250
ELK Stack configurations can range from as low as $30 in the cloud to thousands of dollars based on the requirements. Such distributions can occupy dozens of servers, if not hundreds in larger analytical organizations. In the following architecture I have highlighted a larger elastic stack comprising of multiple levels of redundancy.
This configuration might cost over $330, however when the reserved instance rates are applied, they sharply drop to $175.
What Are Reserved Instances?
These instances cannot be dropped easily without incurring a penalty. Reserved instances are contract based instances for 1year or 3years. Reserved instances are the cheapest among the server industry, however you need to abide by the 1year contract agreement.