{"id":495786,"date":"2023-05-09T11:26:12","date_gmt":"2023-05-09T09:26:12","guid":{"rendered":"https:\/\/www.devoteam.com\/success-story\/government-body-automates-logs-from-hundreds-of-sources-in-one-central-location-using-elastic-cloud-enterprise\/"},"modified":"2023-05-09T11:26:12","modified_gmt":"2023-05-09T09:26:12","slug":"government-body-automates-logs-from-hundreds-of-sources-in-one-central-location-using-elastic-cloud-enterprise","status":"publish","type":"success-story","link":"https:\/\/devoteam.info\/uk\/success-story\/government-body-automates-logs-from-hundreds-of-sources-in-one-central-location-using-elastic-cloud-enterprise\/","title":{"rendered":"Government body automates logs from hundreds of sources in one central location using Elastic Cloud Enterprise"},"content":{"rendered":"\n

The challenge<\/h2>\n\n\n\n

The government body in question receives a large amount of information from hundreds of different sources. They wanted to store this information in one central system. To establish a central logging platform, where infrastructure logging, application logging, and authentication logging could be stored centrally, they decided to start with Elastic<\/a> in 2018.<\/p>\n\n\n\n

The solutions<\/h2>\n\n\n\n

Clearly defining the requirements<\/h3>\n\n\n\n

To ensure that the large volume of incoming log sources is processed in a structured and consistent manner, it was important to clearly define the requirements regarding the type of logs and align them with the Elastic Common Schema<\/a> (ECS). The ECS is a shared data model designed to capture log data from various sources in a standardized way. This allows for more efficient log analysis and quicker interpretation and application of insights derived from the logs. As a result, the processing of logs is streamlined, enabling faster and more accurate response to any issues or trends observed in the logs.<\/p>\n\n\n\n

Control of the environment<\/h3>\n\n\n\n

As the platform grows, the required resources must be considered. The government body has everything self-managed in different data centers. Elasticsearch employs the concept of data tiers<\/a>: hot-warm-cold. This means that different resource criteria apply to each tier.<\/p>\n\n\n\n

For the hot tier, the most powerful resources are used with less storage space, while less powerful (less expensive) resources are used for the warm tier, but with more storage capacity. This pattern continues for the cold and frozen tiers.<\/p>\n\n\n\n

With such a scale of the hardware platform, it is important to maintain control over costs. Additionally, there is a desire to have control over licensing costs, which depend on the chosen subscription.<\/p>\n\n\n\n

Automation<\/h3>\n\n\n\n

Large-scale environments, such as the one in this government body, are impossible to maintain manually. Hence, the need arose for a solution to automate<\/a> this process. Through a structured and standardized approach, all resources and variables can be planned, enabling their consistent and reusable usage.<\/p>\n\n\n\n

Since the organization lacked the necessary expertise in Elastic, experts from Devoteam were engaged to assist. Our experts automated and simplified the installation and configuration of Elastic using Ansible and Terraform.<\/p>\n\n\n\n

Knowledge and expertise<\/h3>\n\n\n\n

In addition to Elastic as the core technology, we provide expertise in other technologies such as Kafka, Docker, and Logstash. This was necessary to bring the logging from the source systems to the Elastic platform.<\/p>\n\n\n\n

The benefits<\/h2>\n\n\n\n

By leveraging Devoteam’s expertise in Elastic, there are several benefits for the government body, including:<\/p>\n\n\n\n