Tag: Azure-Function

Introduction On paper, moving data between two systems looks simple. In practice, it rarely is. Halfway through, files go missing, mappings stop lining up, long jobs time out, and duplicate records slip in. When the work is finally done, someone still has to prove that more than a million records arrived where they belong. Our team at MagnusMinds recently faced this exact challenge. The goal was to migrate a large collection of document attachments from an older source environment into a modern business platform. With over one million files, complex connections between records, and no room for errors, a simple file transfer was out of the question. Here is how we broke down the problem, built a reliable setup with Azure Data Factory and Python, and made sure the migration could recover smoothly from any issue. What Is Large-Scale Attachment Migration? Large-scale attachment migration is the controlled transfer of files along with their details and business connections. The goal is to make sure every file lands in the target system linked to the right business record, so it can be easily found and opened. Three key things separate it from a standard file copy: Context travels with the file: Every file belongs to a business record, such as a material, product, or vendor. That connection must stay in place. Details matter as much as the file itself: File names, types, IDs, and dates decide whether the target platform can actually read and use the document. Accuracy must be easy to prove: When handling millions of items, you need clear proof of what succeeded, what failed, and that nothing was lost or duplicated. The Challenge Several technical challenges came together to make this migration difficult: Huge volume: Moving over one million attachment records ruled out manual or half-manual methods from day one. Complex mapping: Documents had to be matched to specific material numbers using CSV files that listed document IDs, file names, file extensions, and material numbers. One material, multiple documents: A single material could have several files attached to it. The system had to support valid multi-file connections without creating unwanted duplicate records. Files on an SFTP server: The actual documents lived on an external SFTP server. Each file had to be found and retrieved as the pipeline ran. Pipeline time limits: Cloud systems have strict run-time limits. Pushing too much data at once causes jobs to time out or stop unexpectedly. Recovering failed batches: If an error occurred halfway through, we needed a way to re-run only the failed records without starting the whole migration over. Platform synchronization: After updating the database, a sync API had to be called so the new files appeared right away in the platform user interface. How Azure Data Factory and Python Solve It We split the work between two core tools with clear roles: Azure Data Factory (ADF) acts as the manager, deciding what runs, in what order, and what happens if something fails. A Python-based Azure Function acts as the worker, reading the mapping files, finding the documents, and preparing database updates. The pipeline runs through eight clear steps: Find mapping files: A GetMetadata activity checks a folder in Azure Blob Storage and makes a list of all CSV mapping files waiting to be processed. Process files one at a time: A ForEach activity picks up each mapping file in order. Processing one file at a time keeps things stable and makes errors easy to pinpoint. Run the Azure Function: For each file, ADF calls the Python Azure Function with a batch size of 100 records. The Function searches the SFTP server, matches files against the CSV rows, and returns three sets of database instructions: one for attachment records, one for the sync API, and one for file names. Update database records: ADF runs the attachment database instructions returned by the Function directly against the target database. Sync only when needed: An IfCondition checks whether there are platform updates to apply. If yes, it calls the platform sync API. If not, the step is skipped cleanly. Update file names: A final database step updates document names and paths in the platform tables. Keep going after a failure: Every step has a fallback path. If an individual file hits an issue, the pipeline logs the error and moves straight to the next file instead of stopping the entire run. Retry temporary errors automatically: If the Azure Function call fails due to a temporary network issue, it retries up to two times automatically, waiting 60 seconds between attempts. Key Features Automatic file discovery: picks up incoming mapping files on its own without needing a manual list. Step-by-step batch execution: handles one CSV at a time with 100 records per Function call to keep runs smooth and safe. Centralized business logic: the Azure Function builds ready-to-run database scripts, keeping all rules in code rather than cluttered pipeline settings. Three-stage database updates: handles attachment records, sync payloads, and file names in separate, safe steps. Smart API calls: triggers the sync API only when there are real changes to send, avoiding wasted calls. Non-stop error handling: built-in fallbacks allow the pipeline to keep processing remaining files even if one file fails. Advantages No manual tracking: Mapping files are found and processed automatically from start to finish. Predictable run times: Batches of 100 records keep memory usage low and prevent jobs from timing out. Runs without constant supervision: Built-in fallbacks mean the pipeline can process thousands of files without needing someone to watch it the whole time. Targeted re-runs: Failed files can be reprocessed on their own without repeating work that already finished successfully. Clear visibility: ADF provides a clear record of every single file and step, making it easy to see what succeeded and what failed. Limitations to Keep in Mind Mapping file quality is very important: The pipeline depends completely on accurate CSV mapping files. Any missing or wrong information in those files will cause errors. External SFTP dependency: If the SFTP server slows down or drops its connection, file lookups will fail and those batches will be skipped. Function updates require a quick code release: Because the business rules live inside the Python Azure Function, adding new file types or conditions requires updating the code. One-by-one processing trades speed for safety: Processing files one at a time is very stable and easy to fix, but it takes longer than running everything at once. Business Impact Migrating more than one million document attachments requires both accuracy and dependability. A single failure should never put days of work at risk. This solution gave the team a reliable, automated way to move records. It transferred data in controlled batches, caught and isolated errors without losing completed work, and kept the business platform updated in real time. The migration finished without manual file hunting or guessing which records had made it across. Conclusion Large-scale data migrations are rarely just about copying files. True reliability comes from designing workflows that respect system limits, protect connections between business records, and recover smoothly when something goes wrong. By combining Azure Data Factory for pipeline management with Python Azure Functions for flexible processing, teams can turn massive, stressful migrations into predictable, everyday operations.

Most Microsoft Fabric projects serve one organization. Ours had to serve many. A cloud cost management (FinOps) software provider asked us to build the data platform behind their product: collect each customer's Azure and AI spending, turn it into one standard format, and serve it to dashboards and an AI assistant. Every customer had to see only their own data, onboarding had to be automatic, and the platform had to grow region by region. We built it entirely on Microsoft Fabric, with a dedicated workspace for every customer. This post walks through the design, the parts that made it work in production, and what we would tell any team building SaaS on Fabric. The challenge (pain points in real operations) A single-tenant Fabric design is straightforward. Multi-tenant SaaS raises harder questions, and each one had to be answered before the first customer went live. Isolation that can be proven: Customers trust you with billing data. "We are careful" is not enough. The platform had to show, automatically, that one customer's reports can never read another customer's data. One codebase, many customers: Every customer needed the same pipelines, the same data model and the same app. Hand-built workspaces drift apart within weeks. Onboarding without a project: Each new customer needed groups, storage, pipelines, a semantic model, an app and credentials. Doing that by hand would turn every sale into a mini-project. Capacity cost: Fabric capacity is billed whether it is busy or idle. Sizing it for the heaviest nightly run all day would waste money. Growth by region: Customers in different geographies expect their data to stay in region, and an incident in one region should not affect the others. What we built (tailored solution, delivered end-to-end) We designed the platform around three ideas: a regional deployment holds everything a set of customers needs, every customer gets their own workspace, and nothing is created by hand. 1. Regional deployments (growth and blast radius) Customers are placed on regional deployments, sometimes called deployment stamps. Each deployment has its own Fabric capacity, Azure Key Vault, Azure Function Apps and a core Fabric workspace. The core workspace holds the shared notebooks and pipelines, plus a Fabric SQL database that acts as the customer registry. The registry lists every customer with their workspace and lakehouse IDs, status and settings. Pipelines never hard-code a customer. They read the registry at run time. Adding a region means standing up a new deployment from the same templates. An issue in one deployment stays inside it. 2. A workspace per customer (isolation by design) Every customer gets a dedicated Fabric workspace containing: Bronze, Silver and Gold lakehouses in a Medallion Architecture A Direct Lake semantic model that holds every business measure The customer-facing Fabric App, built with Microsoft's Rayfin SDK A Fabric Data Agent that answers plain-English cost questions from that customer's model only Access is controlled through Microsoft Entra security groups per customer. Each customer's Azure credentials live in Key Vault and are read at run time, never stored in code. The important part is that isolation is tested. An automated check confirms that each customer's semantic model is bound only to that customer's own Gold lakehouse, and a read-only smoke test runs across every active customer on a deployment. 3. One pipeline run for every customer (one codebase) The transformation logic is written once and runs for everyone. A master pipeline reads the active customers from the registry, and a ForEach activity runs the per-customer pipeline for each of them: Converting Azure, OpenAI, Anthropic and Microsoft Copilot spend into FOCUS, the open standard for cloud cost data Cost allocation to teams, budgets, anomaly detection and savings recommendations Reservations, price sheets and Microsoft 365 usage A semantic model refresh at the end Every notebook and pipeline lives in source control. Processing is incremental: a watermark table records what each customer has already processed, so nightly runs only touch new data. Ingestion that needs customer credentials stays in Azure. Azure Durable Functions call the Cost Management API, Azure Resource Graph, Microsoft Graph and AI vendor usage APIs, and write Delta tables straight into each customer's OneLake. Azure cost export files reach Fabric through OneLake shortcuts, so raw files are never copied. 4. Onboarding as a workflow (one run instead of a project) Onboarding a customer is a single automated workflow. It: Creates the Entra groups Adds the registry entry Creates the workspace and the three lakehouses Stores the customer's credentials in Key Vault Deploys the semantic model, the app and the Data Agent Runs the isolation check Offboarding reverses it. One detail saved us more than once. A semantic model fails to deploy if it references a table that has no data yet. We generate a skeleton schema listing every table and column the model expects, create empty tables from it at onboarding, and backfill new tables into existing customers. A CI test also checks that every column a notebook writes exists in the model, so a schema change cannot break every customer overnight. 5. Capacity that follows the workload (cost control) Each deployment has two capacity sizes: a smaller resting size for daytime reading and a larger size for the nightly processing run. The capacity scales up before the run and scales down as soon as the master pipeline finishes, rather than after a fixed duration. The app also caches semantic model query results against a data-version stamp, which removes repeated identical queries and keeps the capacity free for real work. 6. Customer-hosted delivery (data residency) Some enterprise customers cannot allow their reporting data to sit in a supplier's tenant. For them, processing stays in the provider's tenant and only the finished Gold data is published into the customer's own Fabric tenant. The semantic model and app run there on the customer's capacity, and no copy is left on the provider's side. That deserves its own post, and we will cover how we built and tested it separately. Real-world impact (what changed after rollout) Isolation you can show an auditor: An automated check proves each customer's model reads only their own data, on every deployment. Onboarding in one workflow: A new customer goes from contract to live workspace, model, app and AI assistant without hand-built steps. One codebase for every customer: A fix or feature in a notebook, pipeline or model reaches all customers in the next release. Lower capacity cost: Compute is sized up only while the nightly run is actually running. Room to grow: New regions are new deployments from the same templates, with problems contained to one deployment. What we would tell any team building SaaS on Fabric Decide your isolation unit first. For us it was the workspace. Everything else, from access groups to deployment scripts, followed from that choice. Keep a registry and read it at run time. Pipelines that look up their customer list never need editing when a customer joins or leaves. Treat every Fabric item as code. Notebooks, pipelines, lakehouses and the semantic model all live in Git and deploy the same way to every workspace. Test isolation automatically. A check that runs on every release catches a binding mistake before a customer does. If you are planning a multi-tenant product or a large data platform on Microsoft Fabric, we would be glad to share what we learned. Connect with us to talk through your architecture.

What is Azure Kubernetes Services Aks? Azure Kubernetes Service (AKS) is a fully managed Kubernetes service offered by Microsoft Azure. It allows you to deploy and manage Kubernetes clusters on the Azure cloud platform. Azure Kubernetes Service makes it easier for developers to deploy, manage and scale containerized applications using Kubernetes. In this article, we will delve deeper into Azure Kubernetes Service and look at its features, benefits, and drawbacks. One of the standout features of AKS lies in its role as an enabler for both development and operations teams. By offering a managed environment, AKS allows developers to channel their efforts towards crafting and refining applications without being burdened by the intricacies of infrastructure provisioning and maintenance. Simultaneously, operations teams benefit from the automation and optimization features inherent to AKS, which simplify the deployment and orchestration of containerized workloads.     Azure Kubernetes Service offers several features that make it an attractive option for developers. These features include: Managed Kubernetes: Azure Kubernetes Service is a fully managed Kubernetes service, meaning that Microsoft manages Kubernetes clusters. This includes provisioning, scaling, and upgrading the Kubernetes clusters. Easy Deployment: AKS makes it easy to deploy Kubernetes clusters on Azure. Developers can deploy a cluster with just a few clicks, making it easy to start with Kubernetes in Azure. High Availability: AKS provides high availability for Kubernetes clusters using multiple nodes in different availability zones. This ensures that the cluster is always available, even in a failure. Security: AKS provides security features such as role-based access control (RBAC) and network security groups (NSGs) to secure Kubernetes clusters. Scalability: AKS allows you to scale your Kubernetes cluster up or down based on your application’s workload. Integration: AKS integrates with other Azure services such as Azure Container Registry, Azure Active Directory, and Azure DevOps.Hybrid cloud capabilities: Azure provides hybrid cloud capabilities, enabling organizations to run Kubernetes clusters both on-premises and in the cloud, and easily move applications between the two environments.        Benefits of Azure Kubernetes Services Simplified Deployment: AKS simplifies the deployment of containerized apps by reducing the complexities of managing infrastructure. Developers can easily use familiar tools and workflows to deploy applications, reducing the learning curve associated with container orchestration. Cost-Efficiency: By leveraging AKS, organizations can achieve cost-efficiency through optimized resource utilization. Avoid unnecessary expenses by scaling resources based on demand, ensuring efficient resource allocation. High Availability: It provides high availability by distributing applications across multiple nodes and availability zones. This ensures that applications remain accessible even in the event of node failures or other infrastructure issues. Security and Compliance: AKS incorporates robust security features, including Azure Active Directory integration, role-based access control (RBAC), and network policies. This helps organizations meet their security and compliance requirements while deploying and managing containerized applications.   Cons of Azure Kubernetes Service Vendor Lock-In: AKS is a Microsoft Azure service, meaning you may be locked into the Azure cloud platform if you choose to use AKS. Cost: AKS is a paid service, which can quickly add up if you have large Kubernetes clusters. Limited Control: AKS is a managed Kubernetes service, meaning that Microsoft manages the Kubernetes cluster. This can limit the level of control you have over the underlying infrastructure. Learning Curve: Although AKS removes the complexity of managing Kubernetes clusters, there is still a learning curve associated with deploying and managing containerized applications on Kubernetes.                                            Why Azure Kubernetes Services? One of the main advantages of AKS is its seamless integration with other Azure services. This makes deploying and managing containerized applications on the Azure cloud platform easy. AKS can be used with Azure Container Registry (ACR) to store and manage container images and Azure DevOps to enable continuous containerized application integration and deployment (CI/CD). Azure Kubernetes Service also simplifies Kubernetes deployment. It automates the deployment, scaling, and management of Kubernetes clusters, so developers can focus on building and deploying their applications. AKS provides features such as automatic scaling, self-healing, and rolling updates, which help ensure that applications are always available and up-to-date. Another advantage of AKS is its high availability. AKS uses multiple nodes in different availability zones, ensuring the Kubernetes cluster is always available. It also supports horizontal scaling, which allows the cluster to adjust automatically to changes in demand. Conclusion Overall, AKS stands out for its seamless integration with Azure services, simplified Kubernetes management, high availability, security features, and Microsoft support.AKS provides a powerful platform for deploying and managing containerized applications, making it easier for organizations to adopt Kubernetes and leverage the full potential of containers in the cloud. Businesses already using Azure for their cloud infrastructure should consider using AKS to deploy and manage their containerized applications.

Serverless computing is a widely adopted approach and an extension of the cloud computing model where customers can focus solely on building logic, with the server infrastructure being completely managed by third-party cloud service providers. In Microsoft Azure, serverless computing can be implemented in various ways, one of which is by using Azure Functions. In this blog, we will discuss how to use Azure Functions for serverless computing. Firstly, let us understand the following terms. What Is Serverless Computing? Serverless computing, also known as the Function-as-a-Service (FAAS) approach to building software applications, eliminates the need to manage the server hardware and software by the consumer and be taken care of by third-party vendors. What Are Azure Functions? Azure functions are the serverless solution that provides all the necessary resources to carry out the tasks with minimal lines of code, infrastructure, and cost. The Azure function are a combination of code and event allowing us to write the code in any language. A Step-by-Step Approach For Creating An Azure Function Go to the Azure portal, search for Function App, and select Function App. Create a new Function App and fill in the details accordingly. Basic tab You can select the Runtime stack and version based on your requirements. Here, I am selecting .NET with version 8 and the operating system Windows. Storage You may leave the default values or configure them according to your project requirements. The default values are configured as. Storage account: You may use the existing storage account or create a new account to store your function app. Monitoring Enable the Application insights to monitor the activity. Deployment tab To enable Continuous Integration and Continuous Deployment (CI/CD), you may connect your function app to a repository by authorizing it to GitHub. These are the important things to focus on while creating your function app, you may leave the remaining details as default or customize them according to your requirements. Once you finish configuring your app, you can click the “create” button at the bottom of the page.Now your app will start the process of deployment. Once deployment is done click on go to the resource tab and you will see your function app was created successfully. Now we need to create a function in our function app. As you can see We have various options to choose Visual Studio, VS code, and other editors or CLI. Choose an environment to create your function. I’ve chosen Visual Studio to create my function app. Create an Azure Functions with Visual Studio From the Visual Studio menu, select File > New > Project. In Create a new project, enter functions in the search box, choose the Azure Functions template, and then select Next. Here you can select the function based on your requirements. Here I am selecting Timer trigger function. Then click on the Create button to create a project. You will see that the default Timer trigger function is created. Here I have created one more function called "HTTPTrigger". Here, you can see two JSON files: host.json and local.settings.json. The local.settings.json file stores app settings and settings used by local development tools. Settings in the local.settings.json file are used only when you're running your project locally. When you publish your project to Azure, be sure to also add any required settings to the app settings for the function app. Publish to Azure Use the following steps to publish your project to a function app in Azure. In Solution Explorer, right-click the project and select Publish. In Target, select Azure then Next. Select Azure Function App (Windows) for the Specific target, which creates a function app that runs on Windows, and then select Next. In the Functions instance, You have to select the function that you created on the Azure portal and then click the finish button.  You can see that the publish profile has been added. Now, click on the Publish button to publish the function to Azure. Once the function is published, go to the Azure portal and search for Application Insights. You can find the Application Insights instance with the same name as the function. On the LHS, go to the Transaction search tab under Investigate and click on See all data in the last 24 hours. In the logs, you can see that your function is working properly. Conclusion In a nutshell, Azure functions provide a very precise environment for developers allowing them to more focus on coding rather than then managing infrastructure. This feature plays a key role in building scalable and responsive applications with low cost.