Multi-Hop/Medallion Architecture | in Databricks-Datacloudy

 

In this blog we are going to see about Multi Hop Architecture in brief manner.

Overview of this Page:

SL.No	Topic	Short Description
1	Introduction	Intro to medallion architecture with lakehouse overview
2	Multi-Hop Architecture	Explanation of Multi-Hop Architecture with the layers
3	Key Principles	Few points on key principle of medallion architecture
4	Application	Application of medallion architecture
5	Benefits	Benefits of medallion architecture

Introduction:

• Multi-Hop architecture is nothing but, medallion architecture. That means it is a data design pattern used to logically organize data in a lakehouse, with the goal of  improving the structure and quality of data as it flows through each layer of the architecture.
  
  
• Since we talked about lakehouse here, we need to have a quick definition on lakehouse.
  
  
• In the dynamic landscape of data management, the Lakehouse Platform emerges as a beacon, casting its net wide across the vast data seas. This revolutionary platform seamlessly blends the best of both data lakes and data warehouses, creating a harmonious ecosystem that invites organizations to embark on a transformative journey of data exploration and analytics. 
  
  
• Dont get demotivated after reading the above points 😅 . Follow below points.
  
  
• Let us see the Lakehouse definition given by Databricks community in their official portal 
  
  "A lakehouse is a new data platform architecture paradigm that combines the best features of data lakes and data warehouses".
  
  
  
• That is the disadvantage of Data warehouse is it cannot store unstructured data, and the disadvantage of data lake is it does not ACID compliant. But in Lakehouse , the structured and unstructured data can be stored and it is also ACID compliant that ensures consistency of data read and written by multiple sources.
  
  
• The benefits of lakehouse model is its is very simple, easy to understand and very easy to implement, it enables incremental ETL, has ACID transaction and Time travel, can recreate the tables from raw data at any time.

Multi-Hop Architecture:

Now let us come to the Topic of Multi-Hop Architecture,  the layers of this architecture are

    i) Bronze Zone

    ii) Silver Zone

    iii) Gold Zone

        Multi hop Architecture is also called Medallion Architecture, It is a design pattern of data used to logically organize the data in lakehouse. It improves the structure and quality of data as it flows to each layer of the architecture. The Simple Understandable architecture shown below,

        

Let us see about each layer one by one.

 i) Bronze Zone:

• Bronze is the first layer of Multi hop architecture, here we land all the data from external source system
  
  
• Which means Bronze Zone has Raw Data.
  
  
• The downstream of Bronze zone is Silver Zone.

ii) Silver Zone:

• Silver Zone is the second layer of Multi hop architecture, Silver Zone has filtered, cleansed and conformed data.
  
  
• The Silver layer brings the data from different sources into an Enterprise view and enables self-service analytics for ad-hoc reporting, advanced analytics and ML.
  
  
• The Downstream of Silver Zone is Gold Zone.

iii) Gold Zone :

• Gold zone is the third layer of Multi-hop architecture, Gold Zone holds the Business level Aggregated data.
  
  
• So, it delivers continuously the updated, cleaned, aggregated data to downstream users.
  
  
• The Downstream members where Data Analyst or Data Scientist. Majorly the Downstream reporting tools used are Power Bi and Tableau

Key Principles:

1. Distributed Processing: The fundamental principle of Multi-Hop/Medallion Architecture is the distribution of processing tasks across multiple nodes. This ensures that no single node becomes a bottleneck, enabling parallel processing and faster computation.
   
   
2. Fault Tolerance: The architecture enhances system resilience by incorporating fault-tolerant mechanisms. In the event of a failure at one node, the system can gracefully redirect tasks to alternative nodes, minimizing downtime and ensuring continuous operation.
   
   
3. Scalability: Multi-Hop/Medallion Architecture facilitates seamless scalability. As computational demands increase, additional nodes can be seamlessly integrated into the architecture, allowing the system to scale horizontally and meet growing requirements without a significant overhaul.
   
   
4. Load Balancing: Intelligent load balancing mechanisms are embedded within the architecture to ensure even distribution of tasks among nodes. This prevents resource overutilization on specific nodes and maximizes overall system efficiency.

Application:

Some of the applications of Medallion Architecture are given below,

1. Big Data Processing: In the realm of big data analytics, Multi-Hop/Medallion Architecture shines. By distributing data processing across multiple nodes, it accelerates the analysis of vast datasets, unlocking actionable insights in real-time.
   
   
2. Distributed Computing Systems: Multi-Hop/Medallion Architecture finds applications in distributed computing systems, where tasks are divided into smaller sub-tasks and processed concurrently. This is particularly valuable in scientific simulations, weather forecasting, and simulations of complex systems.
   
   
3. Blockchain Networks: Blockchain, the decentralized ledger technology, often leverages Multi-Hop/Medallion Architecture to enhance the security and efficiency of transactions. The distributed nature of the architecture aligns with the principles of decentralization inherent in blockchain.
   
   
4. Content Delivery Networks (CDNs): CDNs employ Multi-Hop/Medallion Architecture to optimize the delivery of content by strategically placing servers in geographically dispersed locations. This reduces latency and accelerates the retrieval of content for end-users.

Benefits:

Some of the Benefits of Medallion Architecture are given below,

1. Enhanced Performance: The distributed nature of Multi-Hop/Medallion Architecture leads to improved performance by allowing concurrent processing, minimizing latency, and optimizing resource utilization.
   
   
2. Scalability and Flexibility: Organizations can easily scale their systems by adding new nodes, adapting to changing workloads without disrupting operations. This scalability fosters flexibility in accommodating varying computational demands.
   
   
3. Improved Fault Tolerance: With redundant nodes and failover mechanisms, Multi-Hop/Medallion Architecture enhances system reliability, ensuring continued operation even in the face of hardware failures or disruptions.
   
   
4. Resource Efficiency: By distributing tasks intelligently, the architecture prevents resource bottlenecks, ensuring that computing resources are used efficiently and effectively.

Thus, in this blog we saw a crisp and clear detail of  Multi-Hop or Medallion Architecture, its key principle, application and benefits. Hope it is useful. You can also check out the sql, python and cloud topics in this blog. If you like this blog you can follow us by clicking follow button on the right side of this screen.

You can also follow us on the social media by clicking the social media plugin option given above the follow button.

Thank You !!!