adew is a topic the MagnusMinds engineering team writes about from project work: how it is set up, where it breaks, and what we changed in production. This category page lists all 202 articles filed under adew.
Start with “Making Every New Software Project AI-Ready: Essential Repository Files and AI Skills”, “One Button, Two Stores: Shipping Your Flutter App to Play Store and App Store with GitHub Actions”, or “Migrating Over One Million Attachment Records Reliably”. Open a post for the full write-up, or follow another tag or category on the card to a related topic.
Introduction AI-powered development tools such as Google Antigravity, GitHub Copilot, Cursor, and Claude Code are changing how software teams build, maintain, and review applications. However, the effectiveness of these tools depends heavily on the context they receive. When working with an unfamiliar repository, an AI assistant may not understand the application's architecture, technology stack, coding conventions, or business requirements. This can lead to inconsistent implementations, duplicated code, architectural violations, and incomplete code reviews. The solution is to make project-specific knowledge available directly within the repository. This approach benefits not only AI-powered applications but also traditional software projects built with .NET, Angular, React, and other technologies. 1. Essential Files for an AI-Ready Repository A well-structured repository helps developers and compatible AI assistants understand how a project works and how changes should be implemented. The following files provide a practical starting point: README.md – Project Overview Documents the project's purpose, technology stack, prerequisites, setup instructions, build commands, and testing procedures. It helps developers and AI assistants understand how to run and verify the application. AGENTS.md – Development Instructions Provides repository-level guidance for compatible AI coding agents. It can define architectural boundaries, coding practices, shared component usage, and verification requirements. For example, instructions can specify that database operations must follow existing service and repository patterns rather than being introduced directly into controllers. ARCHITECTURE.md – System Design Explains the application's structure, layers, component responsibilities, database interactions, authentication, and external integrations. This helps AI assistants understand where new functionality belongs. CODING-STANDARDS.md – Engineering Guidelines Documents naming conventions, error-handling practices, dependency injection patterns, security requirements, and code organization rules to maintain consistency across the project. Tool-Specific AI Instruction Files Different AI development tools support different instruction mechanisms. GitHub Copilot: .github/copilot-instructions.md Cursor: .cursor/rules/*.mdc or supported AGENTS.md Claude Code: CLAUDE.md Compatible AI agents: AGENTS.md Teams should configure the appropriate files for the tools they use rather than assuming one instruction file works everywhere. 2. Reusable AI Skills for Everyday Development Repository documentation explains what a project is and how it should be developed. AI skills provide reusable instructions for performing specific engineering tasks. The Agent Skills format uses a directory containing a SKILL.md file with YAML metadata and task-specific instructions. Compatible AI tools can discover and use these skills through their supported mechanisms. A project can maintain reusable skills for: Code Generation: Follow existing architectural patterns, reuse shared components, and implement new functionality consistently. Code Review: Identify potential bugs, security risks, performance concerns, and architectural inconsistencies. Debugging: Analyse error logs, stack traces, and existing code to identify root causes before suggesting fixes. Testing: Follow project testing conventions, cover relevant edge cases, and execute appropriate test suites. For example, a code-review skill can instruct an AI assistant to examine database queries for potential N+1 patterns, check authorization requirements, and identify missing error handling. These instructions make recurring development tasks more consistent without requiring developers to repeatedly explain the same expectations. 3. How Everything Works Together Consider a developer asking an AI assistant to implement an API endpoint for exporting monthly billing statements as PDF files. The assistant can use: README.md to understand the technology stack and project setup. AGENTS.md to follow repository-level development instructions. ARCHITECTURE.md to identify the appropriate application layers. CODING-STANDARDS.md to follow existing coding conventions. Relevant AI skills to guide implementation, testing, and code review. It can then inspect existing implementations, reuse available utilities, follow established patterns, and prepare the necessary tests. The generated changes still require developer review, testing, and validation before they are approved and merged. 4. Why Make Your Repository AI-Ready? A properly documented repository with reusable AI workflows can provide several practical benefits: Consistent Development: Keeps AI-generated contributions aligned with established engineering practices. Less Repetitive Prompting: Reduces the need to repeatedly explain project requirements and conventions. Faster Onboarding: Helps new developers and AI assistants understand the codebase more efficiently. Better Code Reviews: Provides repeatable checks for potential bugs, security concerns, and architectural issues. Reduced Technical Debt: Encourages reuse of existing components and helps preserve architectural boundaries. Conclusion AI-assisted development is not just about choosing the right coding assistant. It is also about giving that assistant the right project context and clear engineering instructions. By maintaining essential repository documentation, configuring tool-specific instructions, and introducing reusable AI skills, development teams can build a stronger foundation for consistent and efficient software engineering. Making a repository AI-ready does not require building an AI application. It starts with organising existing project knowledge so developers and AI assistants can work together more effectively. Build Better Software with MagnusMinds At MagnusMinds IT Solution, we combine modern software engineering practices with AI-assisted development workflows to help businesses build scalable and reliable software solutions. Connect with MagnusMinds to explore how structured engineering practices and AI-powered development can improve your software delivery process.
One Button, Two Stores: Shipping Your Flutter App to Play Store and App Store with GitHub Actions and Fastlane How we went from “whose laptop has the keystore?” to a single Run workflow button that tests, builds, signs and uploads our Android and iOS apps, without anyone on the team owning a Mac. The problem We build a Flutter app for Android and iOS. Like most teams, we started out releasing it by hand, and every release was slow and a little risky. The biggest blocker was iOS. To build an iOS app and upload it to the App Store or TestFlight, you need Xcode, and Xcode only runs on a Mac. Every iOS release meant finding a Mac, installing a distribution certificate and a provisioning profile, and hoping none of them had expired since the last time. Android was easier to build but had its own problems: The upload keystore lived on one or two laptops. If that person was on leave, nothing shipped. Someone had to bump the build number in pubspec.yaml by hand. Sometimes two people uploaded the same number and Play rejected the second one. Every so often a “quick test build” went out pointing at the wrong API. We wanted releases to be boring: repeatable, one click, the same steps every time, with no secrets on anybody’s laptop and no Mac on anybody’s desk. What we ended up with A single manually-triggered GitHub Actions workflow with three inputs: Input Choices Meaning platform android, ios, both Which app(s) to release api dev, prod Which backend the build talks to version Text such as 0.1.5 (optional) The version name shown in the stores and the app; empty uses pubspec.yaml Anyone on the team presses Actions → Mobile release → Run workflow, from any computer. About 15 to 25 minutes later: Android: a signed App Bundle is released to Google Play’s Internal testing track, and testers can install it straight away. iOS: a signed build is processing in TestFlight, and internal testers get it automatically. Why GitHub Actions and Fastlane? We split the work between two tools, each doing what it’s best at. Fastlane handles the how: talking to Google Play and App Store Connect, working out version numbers, running the Flutter build, signing and uploading. It runs the same way on a laptop as it does on CI, so we can still debug a release locally. GitHub Actions handles the where and when: which machine runs the build (Linux for Android, macOS for iOS), which secrets are injected, who has to approve a production build, and keeping the built files for later. That split keeps the workflow file short and easy to read. iOS without a Mac and without Xcode This was the part we were most worried about, and it ended up being the cleanest. GitHub provides hosted macOS machines that already have Xcode installed. Our iOS job simply runs on one of them, so the Mac is GitHub’s problem, not ours. Signing was the other half. The traditional approach is to export a certificate (.p12) and a provisioning profile, store them as secrets, and install them on every run. That’s a lot of moving parts, and certificates expire every year. Instead we use automatic signing with an App Store Connect API key. The only iOS secrets are the API key itself and our Apple Team ID. During the build, Xcode uses that key to create or reuse the distribution certificate and App Store profile on the fly. There are no certificate files to store, renew or lose. Two small settings make iOS releases even smoother: The app’s Info.plist declares that it only uses standard encryption (HTTPS), so TestFlight doesn’t stop each upload to ask about export compliance. The workflow doesn’t wait for Apple to finish processing the build, which takes 10 to 30 minutes. There’s no reason to pay for a macOS machine to sit and watch that. Every release follows the same four stages Android and iOS run the same shape of pipeline, so once you understand one you understand both: Stage What happens Test flutter analyze and flutter test. Any failure stops everything; nothing is built or uploaded. Version Ask the store for the latest build number and add 1. The version name comes from the version input, or pubspec.yaml when it’s empty. Build A clean release build with the right version and API URL baked in. Upload Android goes to Play Internal testing. iOS goes to TestFlight. Key decisions The store decides the build number Keeping the build number in a file means someone has to remember to bump it, and two branches can bump it to the same value. So we stopped trusting the file. Before each build, Fastlane asks Google Play (or TestFlight) for the highest build number it has ever seen and adds one. Nothing is written back to the repo, so the workflow only ever needs read access. Type the version once, see it everywhere The optional version input sets the version name in one go. Type 0.1.5 and it shows up in Play Console, App Store Connect, TestFlight and on the app’s own profile screen. Only a strict format like 0.1.5 is accepted, so a typo fails in seconds instead of after a 20-minute build. Leave it empty and the version from pubspec.yaml is used. Release to testers directly, keep production human Android builds go straight to the Internal testing track instead of waiting as drafts, and iOS builds go straight to TestFlight. Only the app itself is uploaded; the store listing, screenshots and descriptions are never touched. Going live stays a human decision. Builds that use the production API pause until an approved reviewer clicks Approve in GitHub, and promoting a build to Production or submitting it for App Store review is still done by hand, after someone has tested it. Secrets you need Everything sensitive lives in Settings → Secrets and variables → Actions. Nothing is committed to the repo and nothing is needed on a developer’s machine. Secret Platform What it is ANDROID_KEYSTORE_BASE64 Android The upload keystore, base64-encoded ANDROID_KEYSTORE_PASSWORD Android Keystore password ANDROID_KEY_ALIAS Android Key alias ANDROID_KEY_PASSWORD Android Key password PLAY_STORE_JSON_KEY Android Google Cloud service-account JSON key APPLE_TEAM_ID iOS 10-character Apple Developer team ID ASC_KEY_ID iOS App Store Connect API Key ID ASC_ISSUER_ID iOS App Store Connect API Issuer ID ASC_KEY_P8_BASE64 iOS The AuthKey_XXXX.p8 file, base64-encoded The dev and production API URLs are stored as plain variables, since they aren’t secret. One-time store setup The workflow is the easy part. Most of the time goes into store-side setup, and you only do it once. Google Play Upload the first build by hand. The Play API can only upload to an app that already has at least one release. Enable the Google Play Android Developer API in any Google Cloud project. Create a service account and download a JSON key. Invite the service account’s email in Play Console under Users and permissions, and allow it to release to testing tracks. Wait about five minutes for the permissions to take effect. Don’t generate a new keystore: Play only accepts builds signed with the upload key it already knows. Apple Join the Apple Developer Program for your organisation. Create the app in App Store Connect with the right bundle ID. Create an App Store Connect API key with the Admin role, so it can create certificates and profiles. Download the .p8 file straight away, because Apple only lets you download it once. Create an internal TestFlight group and add your testers. The workflow file Save this as .github/workflows/mobile-release.yml: name: Mobile release on: workflow_dispatch: inputs: platform: description: App to release required: true type: choice options: [android, ios, both] default: android api: description: API the build talks to (prod waits for approval) required: true type: choice options: [dev, prod] default: dev version: description: Version shown in the stores, e.g. 0.1.5 (empty = pubspec.yaml) required: false type: string default: '' permissions: contents: read env: FLUTTER_VERSION: 3.47.7 RUBY_VERSION: '3.3' APP_VERSION_NAME: ${{ inputs.version }} jobs: test: name: Analyze and test runs-on: ubuntu-latest defaults: run: working-directory: apps/mobile steps: - uses: actions/checkout@v4 - name: Check the version input if: inputs.version != '' run: | if ! [[ "$APP_VERSION_NAME" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then echo "::error::version must look like 0.1.5, got '$APP_VERSION_NAME'" exit 1 fi - uses: subosito/flutter-action@v2 with: channel: stable flutter-version: ${{ env.FLUTTER_VERSION }} cache: true - run: flutter pub get - run: flutter analyze - run: flutter test android: name: Android ? Play Internal testing needs: test if: inputs.platform != 'ios' runs-on: ubuntu-latest concurrency: group: mobile-release-android cancel-in-progress: false environment: ${{ inputs.api == 'prod' && 'production' || 'development' }} env: SKIP_FLUTTER_TESTS: 'true' APP_PLAY_RELEASE_STATUS: completed APP_DEV_API_URL: ${{ vars.APP_DEV_API_URL }} APP_PROD_API_URL: ${{ vars.APP_PROD_API_URL }} APP_KEYSTORE_PASSWORD: ${{ secrets.ANDROID_KEYSTORE_PASSWORD }} APP_KEY_ALIAS: ${{ secrets.ANDROID_KEY_ALIAS }} APP_KEY_PASSWORD: ${{ secrets.ANDROID_KEY_PASSWORD }} steps: - uses: actions/checkout@v4 - uses: actions/setup-java@v4 with: distribution: temurin java-version: '17' - uses: subosito/flutter-action@v2 with: channel: stable flutter-version: ${{ env.FLUTTER_VERSION }} cache: true - uses: ruby/setup-ruby@v1 with: ruby-version: ${{ env.RUBY_VERSION }} bundler-cache: true working-directory: apps/mobile - name: Restore the upload keystore and the Play key env: KEYSTORE_BASE64: ${{ secrets.ANDROID_KEYSTORE_BASE64 }} PLAY_JSON: ${{ secrets.PLAY_STORE_JSON_KEY }} run: | echo "$KEYSTORE_BASE64" | base64 --decode > "$RUNNER_TEMP/upload-keystore.jks" printf '%s' "$PLAY_JSON" > "$RUNNER_TEMP/play-store.json" echo "APP_KEYSTORE_PATH=$RUNNER_TEMP/upload-keystore.jks" >> "$GITHUB_ENV" echo "APP_PLAY_STORE_JSON_KEY=$RUNNER_TEMP/play-store.json" >> "$GITHUB_ENV" - name: Build and release to Internal testing working-directory: apps/mobile/android run: bundle exec fastlane ${{ inputs.api == 'prod' && 'internal' || 'dev' }} - uses: actions/upload-artifact@v4 with: name: android-${{ inputs.api }}-${{ inputs.version && format('{0}-', inputs.version) || '' }}${{ github.run_number }} path: apps/mobile/build/app/outputs/bundle/release/app-release.aab retention-days: 14 - name: Remove the keystore and key if: always() run: rm -f "$RUNNER_TEMP/upload-keystore.jks" "$RUNNER_TEMP/play-store.json" ios: name: iOS ? TestFlight needs: test if: inputs.platform != 'android' runs-on: macos-latest concurrency: group: mobile-release-ios cancel-in-progress: false environment: ${{ inputs.api == 'prod' && 'production' || 'development' }} env: SKIP_FLUTTER_TESTS: 'true' APP_DEV_API_URL: ${{ vars.APP_DEV_API_URL }} APP_PROD_API_URL: ${{ vars.APP_PROD_API_URL }} APPLE_TEAM_ID: ${{ secrets.APPLE_TEAM_ID }} ASC_KEY_ID: ${{ secrets.ASC_KEY_ID }} ASC_ISSUER_ID: ${{ secrets.ASC_ISSUER_ID }} ASC_KEY_P8_BASE64: ${{ secrets.ASC_KEY_P8_BASE64 }} steps: - uses: actions/checkout@v4 - uses: maxim-lobanov/setup-xcode@v1 with: xcode-version: latest-stable - uses: subosito/flutter-action@v2 with: channel: stable flutter-version: ${{ env.FLUTTER_VERSION }} cache: true - uses: ruby/setup-ruby@v1 with: ruby-version: ${{ env.RUBY_VERSION }} bundler-cache: true working-directory: apps/mobile - name: Build, sign and upload to TestFlight working-directory: apps/mobile/ios run: bundle exec fastlane ${{ inputs.api }} - uses: actions/upload-artifact@v4 with: name: ios-${{ inputs.api }}-${{ inputs.version && format('{0}-', inputs.version) || '' }}${{ github.run_number }} path: apps/mobile/build/ios/ipa/App.ipa retention-days: 14 How the workflow works, in short Inputs: when you click Run workflow, you choose the platform (Android, iOS or both), the API (dev or prod) and, optionally, the version number. permissions: the workflow can only read the repo. It never commits anything back. test job: checks the version format, then runs flutter analyze and flutter test once on Linux. If anything fails, the build jobs never start. The build jobs set SKIP_FLUTTER_TESTS so the tests don’t run again on the more expensive machines. android job: runs on Linux. It decodes the keystore and Play key from secrets into a temporary folder, then Fastlane builds, signs and releases the app to Play Internal testing. The last step deletes the keystore and key, even if the build failed. ios job: runs on GitHub’s macOS machine with the latest stable Xcode. Fastlane builds the app, signs it using the App Store Connect API key and uploads it to TestFlight. This is the job that replaces your own Mac. environment: builds using the prod API run in the production environment, which waits for an approved reviewer. Dev builds run straight away. concurrency: only one upload per store runs at a time. A second run waits its turn, so two releases can never pick the same build number. upload-artifact: keeps a copy of the .aab and .ipa for 14 days, named with the API, version and run number. Troubleshooting These are the errors we actually hit, and what fixed them: Message Fix The caller does not have permission Invite the service account in Play Console and grant app permissions, then wait about 5 minutes. Could not read any versionCode from Play The service account can’t see the app, or no build was ever uploaded by hand. Version code X has already been used Two uploads raced. The concurrency group prevents this on CI; just rerun. The Android App Bundle was signed with the wrong key The keystore secret isn’t the upload key Play expects. iOS export fails after a successful archive Put the signing settings in the Xcode project instead of passing them as build arguments. What we got out of it Anyone on the team can ship, from Windows, Mac or Linux, with nothing installed locally. No Mac and no Xcode needed for iOS releases, and no certificate files to renew. No secrets on laptops. Keys exist only as encrypted GitHub secrets and on a build machine for the length of a job. No more build number clashes, because the store decides and uploads run one at a time. Tests always gate a release. A red test means nothing is built. People still make the go-live decision for production. If you’re running a Flutter app with a small team and still releasing from someone’s laptop, this setup takes about a day, and most of that day is store-console setup you only do once. After that, every release is one button. Stack: Flutter 3.47, Fastlane 2.228, Ruby 3.3, GitHub Actions (ubuntu-latest and macos-latest).
Introduction On paper, moving data between two systems looks simple. In practice, it rarely is. Halfway through, files go missing, mappings stop lining up, long jobs time out, and duplicate records slip in. When the work is finally done, someone still has to prove that more than a million records arrived where they belong. Our team at MagnusMinds recently faced this exact challenge. The goal was to migrate a large collection of document attachments from an older source environment into a modern business platform. With over one million files, complex connections between records, and no room for errors, a simple file transfer was out of the question. Here is how we broke down the problem, built a reliable setup with Azure Data Factory and Python, and made sure the migration could recover smoothly from any issue. What Is Large-Scale Attachment Migration? Large-scale attachment migration is the controlled transfer of files along with their details and business connections. The goal is to make sure every file lands in the target system linked to the right business record, so it can be easily found and opened. Three key things separate it from a standard file copy: Context travels with the file: Every file belongs to a business record, such as a material, product, or vendor. That connection must stay in place. Details matter as much as the file itself: File names, types, IDs, and dates decide whether the target platform can actually read and use the document. Accuracy must be easy to prove: When handling millions of items, you need clear proof of what succeeded, what failed, and that nothing was lost or duplicated. The Challenge Several technical challenges came together to make this migration difficult: Huge volume: Moving over one million attachment records ruled out manual or half-manual methods from day one. Complex mapping: Documents had to be matched to specific material numbers using CSV files that listed document IDs, file names, file extensions, and material numbers. One material, multiple documents: A single material could have several files attached to it. The system had to support valid multi-file connections without creating unwanted duplicate records. Files on an SFTP server: The actual documents lived on an external SFTP server. Each file had to be found and retrieved as the pipeline ran. Pipeline time limits: Cloud systems have strict run-time limits. Pushing too much data at once causes jobs to time out or stop unexpectedly. Recovering failed batches: If an error occurred halfway through, we needed a way to re-run only the failed records without starting the whole migration over. Platform synchronization: After updating the database, a sync API had to be called so the new files appeared right away in the platform user interface. How Azure Data Factory and Python Solve It We split the work between two core tools with clear roles: Azure Data Factory (ADF) acts as the manager, deciding what runs, in what order, and what happens if something fails. A Python-based Azure Function acts as the worker, reading the mapping files, finding the documents, and preparing database updates. The pipeline runs through eight clear steps: Find mapping files: A GetMetadata activity checks a folder in Azure Blob Storage and makes a list of all CSV mapping files waiting to be processed. Process files one at a time: A ForEach activity picks up each mapping file in order. Processing one file at a time keeps things stable and makes errors easy to pinpoint. Run the Azure Function: For each file, ADF calls the Python Azure Function with a batch size of 100 records. The Function searches the SFTP server, matches files against the CSV rows, and returns three sets of database instructions: one for attachment records, one for the sync API, and one for file names. Update database records: ADF runs the attachment database instructions returned by the Function directly against the target database. Sync only when needed: An IfCondition checks whether there are platform updates to apply. If yes, it calls the platform sync API. If not, the step is skipped cleanly. Update file names: A final database step updates document names and paths in the platform tables. Keep going after a failure: Every step has a fallback path. If an individual file hits an issue, the pipeline logs the error and moves straight to the next file instead of stopping the entire run. Retry temporary errors automatically: If the Azure Function call fails due to a temporary network issue, it retries up to two times automatically, waiting 60 seconds between attempts. Key Features Automatic file discovery: picks up incoming mapping files on its own without needing a manual list. Step-by-step batch execution: handles one CSV at a time with 100 records per Function call to keep runs smooth and safe. Centralized business logic: the Azure Function builds ready-to-run database scripts, keeping all rules in code rather than cluttered pipeline settings. Three-stage database updates: handles attachment records, sync payloads, and file names in separate, safe steps. Smart API calls: triggers the sync API only when there are real changes to send, avoiding wasted calls. Non-stop error handling: built-in fallbacks allow the pipeline to keep processing remaining files even if one file fails. Advantages No manual tracking: Mapping files are found and processed automatically from start to finish. Predictable run times: Batches of 100 records keep memory usage low and prevent jobs from timing out. Runs without constant supervision: Built-in fallbacks mean the pipeline can process thousands of files without needing someone to watch it the whole time. Targeted re-runs: Failed files can be reprocessed on their own without repeating work that already finished successfully. Clear visibility: ADF provides a clear record of every single file and step, making it easy to see what succeeded and what failed. Limitations to Keep in Mind Mapping file quality is very important: The pipeline depends completely on accurate CSV mapping files. Any missing or wrong information in those files will cause errors. External SFTP dependency: If the SFTP server slows down or drops its connection, file lookups will fail and those batches will be skipped. Function updates require a quick code release: Because the business rules live inside the Python Azure Function, adding new file types or conditions requires updating the code. One-by-one processing trades speed for safety: Processing files one at a time is very stable and easy to fix, but it takes longer than running everything at once. Business Impact Migrating more than one million document attachments requires both accuracy and dependability. A single failure should never put days of work at risk. This solution gave the team a reliable, automated way to move records. It transferred data in controlled batches, caught and isolated errors without losing completed work, and kept the business platform updated in real time. The migration finished without manual file hunting or guessing which records had made it across. Conclusion Large-scale data migrations are rarely just about copying files. True reliability comes from designing workflows that respect system limits, protect connections between business records, and recover smoothly when something goes wrong. By combining Azure Data Factory for pipeline management with Python Azure Functions for flexible processing, teams can turn massive, stressful migrations into predictable, everyday operations.
documents flow into a semantic retrieval layer that supplies context to a generative model, backed by an ASP.NET Core API and PostgreSQL with pgvector. Why RAG matters for enterprise knowledge Most organizations already hold the answers to their employees’ questions. They sit in policy documents, wikis, handbooks and runbooks. The hard part is putting the right paragraph in front of the person asking, in the words they used to ask. Three approaches handle this differently: Keyword search matches terms. It works when the user knows the document’s vocabulary, and struggles when they ask about “probation” and the policy says “trial period”. A plain generative API call sends the question straight to a model. The answer reads fluently, but the model only knows its training data and the prompt. It has never seen your leave policy, so it will decline or guess. Retrieval-Augmented Generation (RAG) retrieves relevant passages first and gives them to the model together with the question. The original RAG paper describes combining a model’s parametric memory with a non-parametric memory: a dense vector index reached through a retriever. RAG adds a retrieval step between the question and the model. It can improve access to relevant information; it does not guarantee accuracy. Not every application needs RAG. If answers don’t depend on private or changing documents, a direct model call is simpler. RAG earns its place when answers must come from a specific body of documents. The proof of concept To understand the pipeline end to end, I built a small proof of concept called the Enterprise AI Knowledge Assistant. It uses ASP.NET Core for the API, Google’s Gemini API for embeddings and answer generation, and PostgreSQL with the pgvector extension, hosted on Supabase, as the vector store. The knowledge base is deliberately tiny: a text file of five company-policy statements covering annual leave, remote work, password rotation, probation and deployment approval. That is enough to see every stage of RAG working on questions like “How many days of paid leave do employees get?”. It is a learning project, so this article makes no claims about real users, accuracy or performance. What it does show is the workflow, which is the same whether the knowledge base holds five sentences or five thousand pages. The RAG workflow at a glance Two workflows share one retrieval store. Every RAG system has two phases that meet in a vector store. Phase 1, knowledge preparation (done ahead of time). Read the documents, split them into chunks, turn each chunk into a vector with an embedding model, and store the text, vector and source name. Phase 2, question answering (done per question). Turn the question into a vector with the same model, find the stored chunks closest to it, place those chunks in a prompt, and ask the generative model to answer from them. The rest of this article follows those steps in order. Step 1: Chunking, deciding what the model gets to see Embedding a whole handbook as one vector would blur dozens of topics together, and long passages inflate prompt size and cost. Documents are split into chunks small enough to be about one thing and large enough to make sense alone. The usual choices: Fixed-size splitting is simple and predictable but can cut through sentences. Paragraph-based splitting follows the author’s structure but produces uneven sizes. Overlap repeats a little text across chunk boundaries so a fact that straddles two chunks survives intact in at least one. Metadata, such as the source filename, lets you trace a chunk back to its document. The proof of concept splits on paragraphs and groups whole paragraphs into a chunk until a character limit is reached, keeping each statement intact. The demo file becomes three chunks. Each chunk is stored with its source filename. For longer, messier documents, overlap and token-aware sizing are the natural refinements. Step 2: Embeddings, turning text into vectors An embedding model maps text to a list of numbers so that texts with similar meaning end up near each other. That is what lets “how long is probation?” find “new employees have a probation period of six months” without sharing many words. An embedding does not answer anything. It only helps find candidates, and the nearest vectors can still be irrelevant. The proof of concept calls Gemini’s gemini-embedding-001 model and requests 768-dimensional vectors. According to the Gemini embeddings documentation, this model outputs 3072 dimensions by default and supports smaller sizes such as 768, and the docs also describe retrieval task types (RETRIEVAL_DOCUMENT for stored text, RETRIEVAL_QUERY for questions) that suit RAG. The same page names a newer model, gemini-embedding-2, which is worth evaluating. One rule matters above the rest: documents and questions must be embedded with the same model and dimensionality. Vectors from different models are not comparable. Step 3: Storing and searching vectors in PostgreSQL PostgreSQL with pgvector keeps chunks, metadata and embeddings in one familiar database. Each row in a document_chunks table holds the chunk text, its source file, and its embedding in a vector(768) column. Retrieval is a nearest-neighbor query. In pgvector, <=> is cosine distance, and cosine similarity is 1 - distance according to the pgvector reference. The proof of concept orders chunks by cosine distance to the question vector and keeps the top K: .Select(x => new { x.Id, x.Content, x.SourceFile, Distance = x.Embedding.CosineDistance(queryVector) }) .OrderBy(x => x.Distance) .Take(topK) Choosing K is a trade-off. Too small and the answer may miss a relevant chunk; too large and the prompt fills with noise. The demo uses a small K, which suits its small knowledge base. Without an index, pgvector performs exact nearest-neighbor search by scanning the table, which gives perfect recall (same reference). That is ideal at demo scale. For large collections, Supabase’s vector index guide generally recommends HNSW, and EDB’s pgvector documentation describes it as faster with better recall than IVFFlat, at the cost of memory and build time. That is a decision to make when the data grows, not before. Step 4: Building the augmented prompt and generating the answer This is the step that makes the pipeline RAG. The retrieved chunks are formatted into labelled blocks and placed in the prompt ahead of the question: Context: [Source 1] File: mock.txt Employees are entitled to 12 days of annual paid leave every year. Question: How many days of paid leave do employees get? A system instruction tells Gemini to answer using only the Context, not to fill gaps from general knowledge, and to reply with a fixed “not found” sentence when the context lacks the answer. Compare this with the plain call, where only the question is sent and the model answers from its training data. With retrieval, the model can answer from content it never saw in training, such as a company’s own leave policy. Changing the knowledge means updating the stored chunks, not retraining a model. The API response also returns the retrieved passages with their source file, so a reader can see what the answer was based on. How far can you trust a RAG answer? RAG improves the odds of a relevant answer. It does not guarantee a correct one. Grounding by prompt is an instruction, not a guarantee. A model told to use only the context can still misread it or lean on memory. Retrieval quality sets the ceiling. Top-K returns the closest chunks even when none is truly relevant, which is why production systems often add a similarity threshold or a “no good match” path. Returned sources are not verified citations. They show which passages the model was given, not which passage supports which claim. Evaluating a RAG system means testing retrieval and generation separately: did the right chunk come back, and did the answer stay faithful to it? Taking RAG beyond a proof of concept A few concerns move to the front when real documents and real users arrive: Permission-aware retrieval. Access control has to be applied in the retrieval query, before private content reaches the model. OWASP’s Vector and Embedding Weaknesses entry warns about weak access controls and context leakage between groups in shared vector stores. Untrusted documents. Retrieved text is input, and a document can contain instructions. OWASP’s prompt injection guidance notes that RAG does not fully prevent prompt injection and suggests constraining model behavior and clearly separating untrusted content. More sources and formats. PDFs, Word files, wikis and uploads need text extraction and a way to update or remove documents. Measurement. A set of real questions with known answers, used to tune chunking and K. Where this could help When an organization’s answers live in documents, RAG may reduce the friction of finding them: people ask in their own words and get an answer tied to passages in specific files. Those are potential benefits, not measured results. Building this proof of concept showed me that the model call is the smallest part of the system. Useful AI applications depend on how knowledge is chunked and retrieved, who is allowed to see what, and how answers are checked. How could your team make organizational knowledge easier to reach without sacrificing security or reliability? References Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, arXiv:2005.11401. Google, Embeddings, Gemini API, Google AI for Developers. Supabase, Vector indexes, Supabase Docs. pgvector 0.8.2 reference, pgvector documentation (PDF), hosted by Crunchy Data. EnterpriseDB, Using pgvector, EDB Docs. OWASP, LLM08:2025 Vector and Embedding Weaknesses, OWASP Gen AI Security Project. OWASP, LLM01:2025 Prompt Injection, OWASP Gen AI Security Project.
After php2shell: why we build on .NET Two critical PHP flaws were patched in October 2026, and attacks started before the patch shipped. Here is what they teach about choosing a runtime, and why modern .NET is our choice for applications that handle money and personal data. What happened On 21 October 2026 the PHP project released emergency fixes for two vulnerabilities in the PHP interpreter itself, now widely called php2shell: CVE-2026-109618: memory corruption while PHP handles an out-of-memory condition. CVE-2026-109619: a use-after-free in which state from one request survives into later requests served by the same worker process. Both are rated CVSS v4.0 9.2. In the worst case, either one lets an unauthenticated attacker run code on the server with the web worker's privileges. The fixed releases are PHP 8.5.12, 8.4.27, 8.3.36 and 8.2.35. End-of-life branches get no official release, though the PHP security team published minimal source backports for PHP 7.4. Attacks started before the patch. Details leaked on 20 October, a day before the scheduled release. Amazon Threat Intelligence reported exploitation attempts from 17 October, and watchTowr has seen exploitation since the 21st. Several public proof-of-concept exploits exist. Rapid7 notes that none is broadly reliable yet, because each targets a specific OS, PHP version and web server combination. If you run PHP, stop reading and patch. Updating WordPress, Laravel or any other application does not fix this, because the flaw is in the runtime underneath them. Upgrade PHP itself, or confirm that your host has. This isn't a pile-on The PHP security team shipped fixes across four supported branches under the pressure of a leak, and backported to an end-of-life branch as well. That is a good response. Every serious platform ships critical CVEs, .NET included: in October 2025, ASP.NET Core's Kestrel server was patched for an HTTP request-smuggling flaw rated CVSS 9.9 (CVE-2025-55315). So the useful question isn't which platform has zero vulnerabilities. None does. The useful questions are: Which kinds of bugs does the platform make unlikely in the first place? How predictably do fixes arrive, and how long is each version supported? How easily can your team find out it's exposed and roll out the fix? On all three, modern .NET is a strong choice for long-lived business applications. 1. Memory safety removes whole classes of bugs Both php2shell flaws are memory-safety bugs: corruption and use-after-free in native C code. These are the most dangerous kinds of bug, because they let an attacker turn a crash into code execution. C# and the .NET runtime are managed by default. The garbage collector owns object lifetimes, array and string accesses are bounds-checked, and the type system stops you treating one kind of object as another. Ordinary application code cannot free an object and then keep using it. Raw pointers exist only inside explicitly marked unsafe blocks, which are easy to find and review. The runtime still contains native code, and it can still have bugs. But the share of your stack where a single mistake becomes remote code execution is much smaller. 2. Predictable releases and support windows .NET follows a published annual schedule. A new major version ships every November. Even-numbered releases are Long-Term Support (LTS) with three years of patches; odd-numbered releases are Standard-Term Support (STS), now with two years. Version Type Released Supported until .NET 8 LTS Nov 2023 10 Nov 2026 .NET 9 STS Nov 2024 10 Nov 2026 .NET 10 LTS Nov 2025 Nov 2028 Security fixes ship on Patch Tuesday, the second Tuesday of every month, through the Microsoft Security Response Center. You can plan maintenance windows a year ahead and know when an upgrade is due. If you're on .NET 8 or 9 today: both go out of support on 10 November 2026, a few weeks from now. Plan the move to .NET 10 LTS. Moving between modern .NET versions is usually a target-framework change plus a short list of documented breaking changes. 3. Knowing you're exposed, and patching quickly Vulnerable-package warnings on every restore. Since .NET 8, dotnet restore checks NuGet packages against the GitHub Advisory Database and warns on known vulnerabilities, including transitive ones. In .NET 10 it audits all packages by default. One command to list exposure: dotnet list package --vulnerable --include-transitive The runtime is a package you control. Self-contained deployments and container images pin an exact runtime version, so patching is an image rebuild that your CI pipeline can test, not a change to a shared server. Automatic patching on Azure App Service. Framework-dependent apps on App Service receive patched runtimes from the platform. Beyond security: what else you get Performance ASP.NET Core with the Kestrel server consistently ranks near the top of independent web framework benchmarks. More requests per server means lower hosting costs. Runs anywhere Windows, Linux and macOS; x64 and ARM64; containers and Kubernetes. Official slim and "chiseled" container images keep the attack surface small. Native AOT Compile ahead of time to a single native binary with fast startup and a small memory footprint. A good fit for serverless functions and sidecars. Secure defaults built in HTTPS redirection, HSTS, anti-forgery tokens, rate-limiting middleware, the Data Protection API and ASP.NET Core Identity ship in the box rather than as third-party add-ons. Strong typing and tooling C# catches whole categories of mistakes at compile time, including null-reference warnings. Visual Studio, VS Code and Rider make refactoring large codebases safe. Observability Built-in structured logging, metrics and OpenTelemetry support, plus .NET Aspire for local orchestration and dashboards. Productive APIs Minimal APIs, built-in dependency injection, configuration and OpenAPI generation let a new service go from template to production quickly. Open source, backed long term MIT-licensed and developed in the open on GitHub under the .NET Foundation, with Microsoft funding and supporting it commercially. A note on "modern" .NET Everything above is about modern .NET (formerly .NET Core, now just ".NET 8/9/10"). The older .NET Framework 4.x is still supported as a Windows component and still receives security fixes, but it gets no new features, runs only on Windows, and lacks most of the performance, container and tooling improvements described here. If you have Framework 4.x applications, there's no need to panic. It is worth putting a migration on the roadmap. Getting there without a big-bang rewrite Whether you're starting from PHP or from .NET Framework, a staged migration works better than a rewrite: Patch first. Get onto a supported, patched runtime today. Migration is a project; patching is an emergency. Put a reverse proxy in front. Route traffic through an ASP.NET Core app running YARP, which forwards everything to the existing app at first. Move one route at a time (the "strangler fig" pattern). Rebuild an endpoint in .NET 10, switch its route in the proxy, and repeat until nothing is left to forward. For .NET Framework code, the .NET Upgrade Assistant automates project-file conversion and flags APIs that have no direct equivalent. Gate releases on security. Fail CI on vulnerable packages and rebuild container images on every Patch Tuesday. The takeaway php2shell will be patched and forgotten, and there will be another critical CVE somewhere next month, possibly in .NET. You can't pick a platform that never has vulnerabilities. You can pick one that makes the worst kinds of bug rare, ships fixes on a schedule you can plan around, tells you when you're exposed, and lets you roll out a fix as a routine build. For applications that hold customers' money and personal data, that's why we build on modern .NET.
Microsoft Fabric & the Medallion Architecture: One Platform, Three Layers of Trust How one platform and a three-layer pattern can turn scattered systems into one trusted source of truth. Most data teams run orchestration, storage and reporting in separate tools: three subscriptions, three security setups, and copies of data in between. Every copy is a chance for numbers to drift apart. Microsoft Fabric takes a different approach: one platform, one subscription, one copy of data. Here's a quick tour, and how the Medallion architecture fits a typical scenario. Why Fabric? Fabric brings Data Factory, Data Engineering, Data Science, Data Warehouse, Real-Time Intelligence and Power BI together on OneLake: one logical lake for the whole organization, like OneDrive for your data. Two ideas make this practical: Lakehouse and Warehouse. Files and Delta tables for Spark, or a SQL-first engine for structured serving. Both store Delta in OneLake. Shortcuts. Show data from ADLS, S3, Dataverse or another lakehouse as local tables, without copying it. The Medallion architecture in 60 seconds Don't try to make data perfect in one step. Improve it in stages. Bronze (raw): data lands exactly as received, with full history and load metadata. Silver (clean): deduplicated, typed, with conformed keys. Bad rows are set aside. Gold (business-ready): star-schema models, KPIs and aggregates built for Power BI. Follow one employee record: messy in Bronze (padded ID, text date), one clean typed row in Silver, and part of a headcount KPI in Gold. Each layer is a checkpoint you can trace and rebuild. An example scenario: four systems, three departments Picture an organization on Excel, Dataverse, Salesforce and CoreHR, where HR, Operations and Finance pull data by hand. The result: mismatched IDs, copy-paste, and late, unaudited reports. Here's how you could solve it on Fabric: Ingest into Bronze with Data Factory, using the right connector or API for each source. Clean into Silver with PySpark: fix types, remove duplicates, conform IDs. Model into Gold: star schemas with shared dimensions, one per department. Serve with Power BI through Direct Lake, with row-level security per department. Run it from one master pipeline with retries, alerts and the Monitoring hub. Key takeaways Fabric unifies the data stack. OneLake keeps one copy of data. Medallion builds trust layer by layer. Pipelines, Spark and Power BI make the workflow. Start small: one source, three layers. Pick one source in your own environment, decide which Bronze table it becomes, and build outward.
If you work with Power BI, you will come across file types like .pbix, .pbip, .pbit and .pbids. They look similar, but each one has a different job. Knowing which is which saves time, avoids confusion, and makes teamwork easier. This guide covers what each file type does and when to use it. It starts with the files you work with directly, then opens up a .pbip project to show the files inside it. Quick overview # Extension What it is Best used for 1 .pbix Everything in one packaged file Everyday building and sharing 2 .pbip Project folder of readable files Git, teamwork, version control 3 .pbit Report template without data Reusing a design for many clients 4 .pbids Saved data connection Quick, consistent source setup 5 .pbiviz Custom visual package Adding specialised charts 6 .tmdl Model definition in plain text Reviewing and editing the model 7 .pbir Report definition and link to the model Editing pages, visuals and filters 8 .bim Older single-file model format Legacy projects and tools Files you work with directly 1. .pbix: the all-in-one report This is the standard Power BI file. The data, model, pages and visuals are all packed into a single file. Best for: day-to-day report building, and sharing or publishing a complete report. Limitation: it is a sealed file. It is hard to see exactly what changed between two versions, and only Power BI Desktop fully understands it. 2. .pbip: the report as a project A Power BI Project (PBIP) saves the same report as a folder of simple, readable text files instead of one sealed file. A small .pbip file opens the project, and the folders keep the model and the report separate. Best for: version control with Git, team collaboration, and reviewing changes. 3. .pbit: the reusable template A template keeps the design and logic but removes the data. When someone opens it, Power BI asks for connection details or parameters and loads the data fresh. Best for: reusing one report design across many clients, teams or tenants, and starting new reports from a proven structure. 4. .pbids: the saved connection This small file stores only how to connect to a data source, such as the server and database name. It contains no data and no report. Best for: giving colleagues a ready-made connection. Opening the file launches Power BI with the source already selected. 5. .pbiviz: the custom visual A .pbiviz file is a package for a custom visual, such as a chart type that Power BI does not include by default. Once imported, it appears alongside the built-in visuals. Best for: adding specialised visuals to your reports, or sharing a visual your team has built. What's inside a .pbip project? When you save a report as a PBIP, Power BI creates one small .pbip file and two folders: one for the semantic model and one for the report. These folders contain several more file types, and each can be opened and edited on its own. 6. .tmdl: the model in plain text TMDL (Tabular Model Definition Language) describes the semantic model, including tables, columns, measures and relationships, as readable text. It holds the structure, not the rows of data. Best for: reviewing model changes line by line, and creating or updating measures and tables. 7. .pbir: the report definition The .pbir file links the report to its semantic model. The report's pages, visuals and filters are stored alongside it in the PBIR format, as a folder of small files rather than one large block. Best for: editing pages and visuals as files, tracing which filters affect which visual, and applying changes consistently. 8. .bim: the older model format BIM stores the entire model in a single JSON file. It was the standard before TMDL. New projects now favour TMDL, because a single large file is harder to read and to compare between versions. Best for: older projects and external tools that still use it. Which one should you use? Building a report on your own: use .pbix. It is the simplest choice. Working in a team or using Git: use .pbip. Delivering the same report to many clients: create a .pbit template. Making setup easier for a colleague: share a .pbids file. Need a chart Power BI doesn't include: import a .pbiviz visual. Reviewing or editing the model inside a project: open the .tmdl files. Editing pages, visuals or filters inside a project: use the .pbir definition. Working with an older project or tool: you will meet .bim. Conclusion Each file type solves a different problem, from simple sharing with .pbix to team-friendly development with .pbip. Choosing the right one makes reports easier to build, review and reuse.
Delivering the same Power BI report to many tenants doesn't have to mean maintaining many semantic models. When the reporting logic is identical and only the data location changes, one parameterized model can serve every tenant. We learned this while building a reconciliation report for a multi-tenant data migration. Every tenant needed the same answer: did all the data arrive at the destination intact? Rather than cloning and editing a model for each tenant, we built one reusable model and moved the tenant-specific details into Power Query parameters. Here's how it works, and why it made onboarding a routine task instead of a development project. The challenge: The report had the same purpose for every tenant: compare source and destination data after migration and identify any discrepancies. However, each tenant had its own schemas. Without parameterization, onboarding a new tenant would require manually updating schema references throughout the model. As more tenants were added, this would create multiple copies of essentially the same model. This also increases the risk of inconsistencies. A fix applied to one model may not be applied to another, and over time the reports can start behaving differently. We wanted a process that was repeatable, consistent, and easy to maintain. The approach: Power Query parameters We added two parameters to the semantic model, and they became the only tenant-specific configuration: SourceSchema identifies the tenant's source schema. DestinationSchema identifies the tenant's destination schema. The server and database connection stay the same for everyone. Only the schema values change. Tenant SourceSchema DestinationSchema Customer A CustomerA_Source CustomerA_Destination Customer B CustomerB_Source CustomerB_Destination Every query in the model references these parameters instead of a hardcoded schema name. Tenant details become configuration, and the reporting logic stays untouched. Onboarding a new tenant in nine steps With the reusable model in place, adding a tenant follows the same checklist every time: Download or prepare the existing semantic model. Rename it for the new tenant. Copy the required report visuals into the new file, if needed. Publish the report to the right Power BI workspace. Open the model's transformation or parameter settings. Set SourceSchema and DestinationSchema to the new tenant's values. Apply the changes. Refresh the model. Validate the reconciliation results. No queries are rebuilt and no report pages are redesigned. The administrator changes two values and validates the output. Consistency, governance, and a word on security One of the biggest advantages of this approach is consistency. Every tenant can use the same semantic model design and reconciliation logic. Tenant-specific values are kept in parameters instead of being scattered throughout the queries. This also makes the process easier to document. The organization can define who is responsible for onboarding tenants, who can change parameters, how refreshes are performed, and what validation is required after each deployment. At the same time, tenant data isolation still needs to be handled through the appropriate database and Power BI security and permission model. Parameters should be treated as configuration, not as the only security control. Business benefits A parameter-driven model pays off in practical ways: Faster onboarding, because there is little to no development per tenant. Consistent reporting, because every tenant shares one model and one set of calculations. Lower maintenance, because a fix is made once and applies everywhere. Fewer configuration errors, because no one is hand-editing dozens of queries. Better scalability, because the effort per tenant stays flat as the tenant count grows. Simpler administration, through a documented, repeatable onboarding process. Above all, adding a tenant becomes an operational task rather than another development project. Conclusion When several tenants need the same Power BI reporting experience, a separate model per tenant usually isn't necessary. If the database structure is consistent and the main difference is the schema, Power Query parameters make one semantic model reusable across all of them. With SourceSchema and DestinationSchema as the only tenant-specific settings, the reconciliation logic stays standardized and onboarding stays repeatable. The result is one model, configurable tenant values, and reporting that scales with your tenant base.
Most Microsoft Fabric projects serve one organization. Ours had to serve many. A cloud cost management (FinOps) software provider asked us to build the data platform behind their product: collect each customer's Azure and AI spending, turn it into one standard format, and serve it to dashboards and an AI assistant. Every customer had to see only their own data, onboarding had to be automatic, and the platform had to grow region by region. We built it entirely on Microsoft Fabric, with a dedicated workspace for every customer. This post walks through the design, the parts that made it work in production, and what we would tell any team building SaaS on Fabric. The challenge (pain points in real operations) A single-tenant Fabric design is straightforward. Multi-tenant SaaS raises harder questions, and each one had to be answered before the first customer went live. Isolation that can be proven: Customers trust you with billing data. "We are careful" is not enough. The platform had to show, automatically, that one customer's reports can never read another customer's data. One codebase, many customers: Every customer needed the same pipelines, the same data model and the same app. Hand-built workspaces drift apart within weeks. Onboarding without a project: Each new customer needed groups, storage, pipelines, a semantic model, an app and credentials. Doing that by hand would turn every sale into a mini-project. Capacity cost: Fabric capacity is billed whether it is busy or idle. Sizing it for the heaviest nightly run all day would waste money. Growth by region: Customers in different geographies expect their data to stay in region, and an incident in one region should not affect the others. What we built (tailored solution, delivered end-to-end) We designed the platform around three ideas: a regional deployment holds everything a set of customers needs, every customer gets their own workspace, and nothing is created by hand. 1. Regional deployments (growth and blast radius) Customers are placed on regional deployments, sometimes called deployment stamps. Each deployment has its own Fabric capacity, Azure Key Vault, Azure Function Apps and a core Fabric workspace. The core workspace holds the shared notebooks and pipelines, plus a Fabric SQL database that acts as the customer registry. The registry lists every customer with their workspace and lakehouse IDs, status and settings. Pipelines never hard-code a customer. They read the registry at run time. Adding a region means standing up a new deployment from the same templates. An issue in one deployment stays inside it. 2. A workspace per customer (isolation by design) Every customer gets a dedicated Fabric workspace containing: Bronze, Silver and Gold lakehouses in a Medallion Architecture A Direct Lake semantic model that holds every business measure The customer-facing Fabric App, built with Microsoft's Rayfin SDK A Fabric Data Agent that answers plain-English cost questions from that customer's model only Access is controlled through Microsoft Entra security groups per customer. Each customer's Azure credentials live in Key Vault and are read at run time, never stored in code. The important part is that isolation is tested. An automated check confirms that each customer's semantic model is bound only to that customer's own Gold lakehouse, and a read-only smoke test runs across every active customer on a deployment. 3. One pipeline run for every customer (one codebase) The transformation logic is written once and runs for everyone. A master pipeline reads the active customers from the registry, and a ForEach activity runs the per-customer pipeline for each of them: Converting Azure, OpenAI, Anthropic and Microsoft Copilot spend into FOCUS, the open standard for cloud cost data Cost allocation to teams, budgets, anomaly detection and savings recommendations Reservations, price sheets and Microsoft 365 usage A semantic model refresh at the end Every notebook and pipeline lives in source control. Processing is incremental: a watermark table records what each customer has already processed, so nightly runs only touch new data. Ingestion that needs customer credentials stays in Azure. Azure Durable Functions call the Cost Management API, Azure Resource Graph, Microsoft Graph and AI vendor usage APIs, and write Delta tables straight into each customer's OneLake. Azure cost export files reach Fabric through OneLake shortcuts, so raw files are never copied. 4. Onboarding as a workflow (one run instead of a project) Onboarding a customer is a single automated workflow. It: Creates the Entra groups Adds the registry entry Creates the workspace and the three lakehouses Stores the customer's credentials in Key Vault Deploys the semantic model, the app and the Data Agent Runs the isolation check Offboarding reverses it. One detail saved us more than once. A semantic model fails to deploy if it references a table that has no data yet. We generate a skeleton schema listing every table and column the model expects, create empty tables from it at onboarding, and backfill new tables into existing customers. A CI test also checks that every column a notebook writes exists in the model, so a schema change cannot break every customer overnight. 5. Capacity that follows the workload (cost control) Each deployment has two capacity sizes: a smaller resting size for daytime reading and a larger size for the nightly processing run. The capacity scales up before the run and scales down as soon as the master pipeline finishes, rather than after a fixed duration. The app also caches semantic model query results against a data-version stamp, which removes repeated identical queries and keeps the capacity free for real work. 6. Customer-hosted delivery (data residency) Some enterprise customers cannot allow their reporting data to sit in a supplier's tenant. For them, processing stays in the provider's tenant and only the finished Gold data is published into the customer's own Fabric tenant. The semantic model and app run there on the customer's capacity, and no copy is left on the provider's side. That deserves its own post, and we will cover how we built and tested it separately. Real-world impact (what changed after rollout) Isolation you can show an auditor: An automated check proves each customer's model reads only their own data, on every deployment. Onboarding in one workflow: A new customer goes from contract to live workspace, model, app and AI assistant without hand-built steps. One codebase for every customer: A fix or feature in a notebook, pipeline or model reaches all customers in the next release. Lower capacity cost: Compute is sized up only while the nightly run is actually running. Room to grow: New regions are new deployments from the same templates, with problems contained to one deployment. What we would tell any team building SaaS on Fabric Decide your isolation unit first. For us it was the workspace. Everything else, from access groups to deployment scripts, followed from that choice. Keep a registry and read it at run time. Pipelines that look up their customer list never need editing when a customer joins or leaves. Treat every Fabric item as code. Notebooks, pipelines, lakehouses and the semantic model all live in Git and deploy the same way to every workspace. Test isolation automatically. A check that runs on every release catches a binding mistake before a customer does. If you are planning a multi-tenant product or a large data platform on Microsoft Fabric, we would be glad to share what we learned. Connect with us to talk through your architecture.