lumen

Big Data Solutions

Solutions based on the Data Lake concept

Features:

  • Support for all types of data
  • Long term storage and access to historical data

  • Data views and slices for all types of users in the organization

  • Support for advanced analytics

Data lake — is a concept of storing data in raw (unprocessed) form, which involves storing data in different schemas and formats at the same time. The most common formats are files and blob objects. The data lake is used as a single repository for all company data, including both raw copies of data generated by corporate systems and derived data (cleaned detailed data, showcases, aggregates) used for tasks such as building reports and dashboards, analytics and machine learning. The data lake can include structured data from relational databases, semi-structured data, syslog, CSV, XML, JSON, unstructured data (emails, documents, PDF files) and binary data (images, audio, video).

Data Lake Solution Architecture

Advantages

  • Cost-effective — Deployable on generic servers

  • Flexibility — easy to add a new data source

  • Scalability — connecting new servers on the fly

  • Reduced time-to-market — fast time-to-market for new products

Data Lake Concept Solution Architecture

Typical Layers.

  • Ingestion Layer — data loading layer using stream and batch technologies

  • Datastore Layer — data storage layer (Data Lake itself)

  • Processing Layer — data processing layer for creating new structures and formats

  • Access Layer — the layer for end users to access and search for data

  • Analytics Layer — layer of analytics and machine learning tools

External systems and data sources.

  • Data Sources — Layer of external data sources (file sources, streaming data, OLTP databases)

  • ETL/ELT — data extraction, conversion and loading tools

  • Data Warehouse — classical enterprise structured data warehouse

  • Business Intelligence — business analytics, visualization, reporting and dashboard tools

For whom solutions based on the Data Lake concept will work:

If your organization is planning to implement or is in the process of implementing a data driven approach (data-driven business decision making) and faces the limitations of current classic data warehouses (no ability to handle unstructured data sources, limited data storage, high cost of ownership, complex storage structure support, long delays in time from connecting a new source to real business value from data use, limited analytical capabilities

Delta Solutions LLC offers a full range of Data Lake implementation services based on Cloudera (Hadoop) products.