What is a data lake?

Data lakes arose to help organizations deal with the flood of big data—much of it unstructured—created by internet-connected apps and services in the late 2000s and early 2010s.  

Unlike traditional databases and data warehouses, data lakes do not require that all data follows one defined schema. Instead, data lakes can store different types of data in varying formats in one centralized repository. Data lakes also take advantage of cloud computing to make data storage more scalable and affordable.

Data lakes are core components of many organizations’ data architectures today. According to the IBM CDO Study, 75% of leading chief data officers (CDOs) are investing in data lakes. 

Thanks to their flexible storage, data lakes can help organizations knock down data silos and build holistic data fabrics. They are also useful for data scientists and data engineers, who often use data lakes to manage the massive unstructured datasets necessary for artificial intelligence (AI) and machine learning (ML) workloads.

For a long time, organizations relied on relational databases (developed in the 1970s) and data warehouses (developed in the 1980s) to manage their data. These solutions are still important parts of many organizations’ IT ecosystems, but they were designed primarily for structured datasets.

With the growth of the internet—and especially the arrival of social media and streaming media—organizations found themselves dealing with a lot more unstructured data, such as free-form text and images. Because of their strict schemas and comparatively expensive storage costs, warehouses and relational databases were ill-equipped to handle this influx of data.  

In 2011, James Dixon, then the chief technology officer at Pentaho, coined the term “data lake.” Dixon saw the lake as an alternative to the data warehouse. Whereas warehouses deliver preprocessed data for targeted business use cases, Dixon imagined a data lake as a large body of data housed in its natural format. Users could draw the data they needed from this lake and use it as they pleased.

Many of the first data lakes were built on Apache Hadoop, an open-source software framework for distributed processing of large datasets. These early data lakes were hosted on-premises, but this quickly became an issue as the volume of data continued to surge.

Cloud computing offered a solution: moving data lakes to more scalable cloud object storage services.

Data lakes are still evolving today. Many data lake solutions now offer features beyond cheap, scalable storage, such as data security and governance tools, data catalogs and metadata management.

Data lakes are also core components of data lakehouses, a relatively new data management solution that combines the low-cost storage of a lake and the high-performance analytics capabilities of a warehouse. (For more information, see “Data lakes vs. data lakehouses”).

While the earliest data lakes were built on Hadoop, the core of a modern data lake is a cloud object storage service. Common options include Amazon Simple Storage Service (Amazon S3), Microsoft Azure Blob Storage, Google Cloud Storage and IBM Cloud Object Storage.

Cloud object storage enables organizations to store different kinds of raw data all in the same data store. It is also generally more scalable and more cost-effective than on-premises storage. Cloud storage providers enable users to spin up large clusters on demand, requiring payment only for the storage used.

Storage and compute resources are separated from one another in a data lake architecture. To process data, users must connect external data processing tools. Apache Spark, which supports interfaces such as Python, R and Spark SQL, is a popular choice.

Decoupling storage and compute helps keep costs down and scalability high. Organizations can add more storage without scaling compute resources alongside it.  

The central data lake storage is connected to various data sources—such as databases, apps, Internet of Things (IoT) devices and sensors—through an ingestion layer.  

Most data lakes use an extract, load, transform (ELT) rather than an extract, transform, load (ETL) process to ingest data. Data remains in its original state when the lake ingests it, and it is not transformed until it is needed. This approach—applying a schema only when data is accessed—is called “schema-on-read.”

In addition to these core components, organizations can build other layers into their data lake architectures to make them safer and more usable. These layers can include:

• Multiple, distinct storage layers to accommodate different stages of data processing. For example, a data lake might have one layer for raw data, one layer for cleansed data and one layer for trusted application data. 

• Security and governance layers, such as integrated data governance solutions or identity and access management (IAM) controls maintain data quality and protect against unauthorized access. 

• A data catalog to help users easily find data by using metadata filters or other methods.

Data lakes, warehouses and lakehouses are all types of data management tools, but they have important differences. They’re often used together in an integrated data architecture to support various use cases.  

Like a data lake, a data warehouse aggregates data from disparate data sources in a single store, usually a relational database system. The key difference is that data warehouses clean and prepare the data they ingest so that it is ready to be used for data analytics.

Data warehouses are primarily designed to support high-performance queries, near real-time analytics and business intelligence (BI) efforts. As such, they are optimized for structured data and tightly integrated with analytics engines, dashboards and data visualization tools.

Warehouses tend to have more expensive, less flexible and less scalable storage than data lakes. Organizations generally use warehouses for specific analytics projects while relying on data lakes for large-scale, multipurpose storage.  

A data lakehouse is a data management solution that combines the flexible data storage of a lake and the high-performance analytics capabilities of a warehouse.

Like a data lake, a data lakehouse can store data in any format at a low cost. Data lakehouses also build a warehouse-style analytics infrastructure on top of that cloud data lake storage system, merging features of the two solutions.  

Organizations can use lakehouses to support numerous workloads, including AI, ML, BI and data analytics. Lakehouses can also serve as a modernization pathway for data architectures. Organizations can slot lakehouses alongside existing lakes and warehouses without a costly rip-and-replace effort.  

Data lakes can help organizations get more value from their business data by making it easier to store, share and use that data. More specifically, data lakes can provide:

• Flexibility: Data lakes can ingest structured, semi-structured and unstructured datasets. Organizations don’t need to maintain separate storage systems for different types of data, which can help simplify data architectures.

• Low costs: Data does not need to go through a costly cleaning and transformation process for storage, and cloud object storage is generally cheaper than on-premises alternatives. Organizations can optimize their budgets and resources more effectively across data management initiatives.

• Scalability: Because data lakes decouple compute and storage resources, and because they typically use cloud storage services, they’re easier to scale up or down than many other data storage solutions. 

• Fewer data silos: According to benchmarking data from the IBM Institute for Business Value, 61% of organizations say that data silos are one of their top challenges. Data lakes can help knock down data silos by removing the need to store different types of data in different places. A central data lake or set of data lakes can be more accessible than disparate data stores spread across business units.  

Because they do not enforce a strict schema and accept many different data types from many sources, data lakes can struggle with data governance and data quality. Without proper management, data lakes can easily become “data swamps”—messy mires of unreliable data that make it hard for users to glean actionable insights.

To combat data swamps, organizations can invest in tagging and classification tools, such as metadata management systems and data catalogs, that make navigation easier. 

Data governance and security solutions, such as access controls, data loss prevention tools and data detection and response solutions, can help ensure that data is not accessed, used or altered without authorization.  

Data lakes do not have built-in processing and querying tools like many warehouses and lakehouses do. Moreover, query and analytics performance can suffer as the volume of data fed into a data lake grows, especially if data is not optimized for retrieval.

Using the right tools and configurations—such as optimized storage formats and query engines—can help ensure high performance, regardless of the data lake's size.