Data Migration

𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝘀: 𝗦𝗲𝗿𝘃𝗲𝗿𝗹𝗲𝘀𝘀 𝗗𝗲𝗹𝘁𝗮 𝗟𝗮𝗸𝗲 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 𝗼𝗻 𝗔𝗪𝗦 =========================================== Imagine you have data in your company's local servers (on-premises) and want to: 1. Move this data to AWS 2. Analyze it without managing servers 3. Use an event-driven approach Here's how TrueBlue, a company facing this challenge, solved it using AWS services: 𝟭. 𝗗𝗮𝘁𝗮 𝗠𝗶𝗴𝗿𝗮𝘁𝗶𝗼𝗻 ----------------- • Used AWS Database Migration Service to copy data from local databases to Amazon S3 • Ensures up-to-date information for jobs, job requests, and workers • Enables accurate job matching 𝟮. 𝗘𝘃𝗲𝗻𝘁-𝗗𝗿𝗶𝘃𝗲𝗻 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 ------------------------------ • Set up S3 event notifications when new data arrives • Used Amazon SQS (Simple Queue Service) to capture these events • Created 3 SQS queues for different update frequencies:  - 10-minute updates  - 60-minute updates  - 3-hour updates • AWS EventBridge rules trigger Step Functions based on these time intervals • Step Functions orchestrate AWS Glue jobs for data processing 𝟯. 𝗦𝗲𝗿𝘃𝗲𝗿𝗹𝗲𝘀𝘀 𝗣𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴 -------------------------- • Chose AWS Glue over Amazon EMR (Elastic MapReduce) for serverless data processing • Reasons for choosing Glue:  - Team's expertise in serverless development  - Easier to manage and debug  - Achieves similar results to EMR without server management • Glue jobs transform and load data into the Delta Lake format 𝟰. 𝗔𝗻𝗮𝗹𝘆𝘁𝗶𝗰𝘀 ------------ • Data scientists use PySpark SQL to query the Delta Lake • Delta Lake has three tiers:  1. Bronze: Raw data from source systems  2. Silver: Cleaned and joined data from bronze tier  3. Gold: Prepared data for machine learning (feature store) • Glue jobs keep the Delta Lake up-to-date with reliable upserts (updates and inserts) • Enables data scientists to:  - Perform accurate job matches  - Extract datasets for analysis  - Build and train machine learning models 𝗕𝗲𝗻𝗲𝗳𝗶𝘁𝘀 𝗼𝗳 𝘁𝗵𝗶𝘀 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲: ------------------------------ 1. Serverless: No need to manage infrastructure 2. Scalable: Can handle increasing data volumes 3. Cost-effective: Pay only for resources used 4. Real-time: Event-driven updates keep data fresh 5. Flexible: Supports various data processing needs This architecture showcases how to build a modern, serverless data lake using AWS services, enabling efficient data migration, processing, and analytics without the complexity of managing servers. #dataengineer #dataengineering #deltalake #aws

The “Before & After” Data Transformation Story In the lead-up to our SAP migration, we weren’t just preparing systems — we were unearthing years of neglected, inconsistent, and chaotic data. If we are honest, most of the time, it felt less like digital transformation and more like an archaeological excavation. We were buried in layers of spreadsheets, conflicting legacy reports, and systems that hadn’t seen a clean-up in over a decade. Each click revealed more clutter: customer names spelled five different ways, address fields mixing “St.” and “Street” like it was a coin toss, duplicate records stacked on top of each other, and critical fields left blank or filled with guesswork. It was more than just messy — it was risky - A complete nightmare! Data was being pulled from everywhere and nowhere. No single source of truth. No consistency. Just a patchwork of outdated inputs fuelling vital business operations. The worst part? We had to tackle it manually. A Time Sink: Highly skilled people stuck doing low-value, repetitive tasks. An Error Magnet: Fatigue set in. Errors crept through. Fix one issue, uncover two more. A Business Risk: Dirty data meant dirty output. Reports couldn’t be trusted. Customers were misbilled. Orders were sent to the wrong place. And confidence in the system? Gone. We knew we couldn’t carry that baggage into SAP. Something had to change. At this point, we built a purpose-specific solution which was created to automate and streamline data cleansing and validation, giving us the ability to: Proactively identify and rectify errors with precision. Ensure data consistency across all records. Validate information against business rules before migration. This impacts business by: 🔹Reducing Pre-Migration Data cleansing and validation Effort by Up to 75% Freeing up SMEs for strategic tasks, cutting contractor costs, and accelerating migration timelines. 🔹Delivering >99% Accuracy in Key Master Data Minimising migration errors, de-risks go-live, building trust in the new SAP system from day one. 🔹Reducing Migration Delays and Rework by 20–40% Fewer surprises in load cycles and UAT, protecting timelines, budgets, and overall project momentum. 🔹Achieving 100% Data Auditability and Compliance Ensuring full traceability, streamlining audits, and providing a defensible position on data quality from day one. 🔹Reducing Post-Go-Live Errors by 15–30% Fewer issues like misbilling and mis-shipments, leading to smoother operations, faster user adoption, and trusted SAP insights. If any of this sounds familiar, you're not alone. The good news is that we have built a solution which has already helped others through their migration journey, and we’d be happy to share it if it’s useful. Just drop us a message. Created in collaboration with Pawel Lipko ↗️

On prem to Cloud migration Step-by-Step AWS Cloud Migration Process 1. Plan the Migration Assessment: Identify the current environment (servers, databases, dependencies, and configurations). Inventory: Document application components and dependencies. Sizing: Determine AWS resources (EC2 instance types, RDS configurations, etc.) based on current usage. Network Design: Plan VPC setup, subnets, security groups, and connectivity. Backup Plan: Create a fallback plan for any issues during migration. 2. Prepare the AWS Environment VPC Setup: Create a VPC with subnets across multiple Availability Zones (AZs). Security: Configure security groups, IAM roles, and policies. Database Configuration: Set up an Amazon RDS instance or EC2-based database for the migration. AD Server: Use AWS Managed Microsoft AD or deploy your AD on EC2. Application Server: Launch EC2 instances and configure the operating system and required dependencies. 3. Migrate Database Backup: Create a backup of the current database. Export/Import: Use database migration tools (e.g., AWS DMS or native database tools) to migrate data to the AWS database. Replication: Set up database replication for real-time sync with the on-prem database. Validation: Verify data consistency and integrity post-migration. 4. Migrate Application Server Packaging: Package the application (e.g., as Docker containers, AMIs, or simple binaries). Deployment: Deploy the application on AWS EC2 instances or use AWS Elastic Beanstalk. DNS Configuration: Update DNS records to point to the AWS environment. 5. Migrate Active Directory (AD) Replication: Create a replica of the on-prem AD in AWS using the AD Trust setup. DNS Sync: Sync DNS entries between on-prem and AWS environments. Validation: Test authentication and resource access. 6. Test and Validate End-to-End Testing: Validate the complete environment (application, database, and AD). Performance Check: Monitor performance using CloudWatch and address any issues. Failover Testing: Simulate failure scenarios to ensure HA/DR readiness. 7. Cutover and Go Live Schedule Downtime: Coordinate with stakeholders and users for a minimal downtime window. Final Sync: Perform a final sync of the database and switch traffic to AWS. DNS Propagation: Update DNS settings to route traffic to the AWS environment (may take up to 24 hours). Monitoring: Continuously monitor AWS resources and performance post-migration. 8. Post-Migration Optimization Scaling: Implement auto-scaling policies for the application. Security: Regularly review and improve security configurations. Cost Optimization: Use AWS Cost Explorer to analyze and optimize resource usage. Downtime Considerations Database Migration: Plan a maintenance window of 2–4 hours for the final database sync and cutover. DNS Propagation: Approx. 15 minutes to 24 hours, depending on TTL settings. Use short TTLs during migration to minimize delays. #AWSMigration #CloudMigration #MinimalDowntime #DatabaseToAWS #ApplicationToAWS #ADToAWS

We almost brought a 20-year-old mistake into S/4HANA. During a recent S/4 migration for a pharma client, "Clean Core" was the mandate from the steering committee. But when we ran the readiness check, the system flagged over 12,000 custom Z-programs. The project timeline was tight. The business sponsor panicked. "Just lift and shift them all," he said. "We can’t risk breaking operations. We will clean up the custom code in Phase 2." If you’ve been in the SAP world long enough, you know the ugly truth: Phase 2 never happens. Instead of arguing, I asked our Basis team to run a simple background job: a 12-month usage report on those 12,000 custom programs. The results were staggering. The Reality Check: Custom objects in the system: 12,000 Objects executed in the last year: 2,400 Objects executed in the last 30 days: 850 They were about to spend hundreds of thousands of dollars and risk the stability of their new S/4 system, just to migrate digital ghosts. Code that belonged to employees who had retired a decade ago. Workarounds for business processes that no longer existed. We didn't just delete the code. We printed the report and put it on the sponsor's desk. The conversation shifted instantly from "How do we migrate this?" to "Why are we hoarding this?" An S/4HANA migration is not an IT infrastructure project. It is a corporate garage sale. If you don't have the courage to throw things away before you move, you aren't transforming. You're just relocating your mess. What is the craziest piece of legacy Z-code you’ve seen someone try to drag into an S/4HANA system?

🚀 Understanding Data Migration the Right Way (SAP S/4HANA Perspective) Data migration is often seen as a technical activity but in reality, it is a business critical transformation process. After working through multiple migration scenarios, one thing is clear: 👉 Successful migration is less about tools, and more about structure, discipline, and data quality. Here is a simple but powerful framework we followed : 🔹 Data Identification – Define scope, ownership, and what really matters. 🔹 Data Extraction – Pull clean, relevant data from source systems. 🔹 Data Mapping – Align source to target (including value mappings). 🔹 Data Cleansing – Fix inconsistencies, remove duplicates 🔹 Data Transformation – Convert into SAP-ready format 🔹 Data Validation – Ensure completeness and correctness 🔹 Data Load – Move data using BAPIs / Migration tools 🔹 Reconciliation – Verify source vs target (most critical step!) 🔹 Business Sign-off – Get stakeholder approval 🔹 Cutover & Final Migration – Go-live with confidence 💡 Key Insight: 80% of migration effort goes into mapping, cleansing, and validation but not the actual data load. A clean migration is not just about moving data, it is about building trust in the new system from Day 1. Sharing a visual flow to simplify this 👇 #SAP #S4HANA #DataMigration #ERP #Consulting #DigitalTransformation #SAPSD #BusinessTransformation

Still using MSSQL RDS as your primary database and paying millions? Ever heard of Babelfish? At Medibuddy, we migrated 7 TB of data from MSSQL to Aurora PostgreSQL and saved 40% in RDS infrastructure costs. We achieved this using Babelfish and AWS DMS, which turned out to be both cost-efficient and migration-efficient. We faced multiple roadblocks along the way, and I wanted to document and share what we learned that might help you too! First, the WHY? 1. MSSQL was hurting us with its costs. 2. We also wanted to unify our database stack and build deep expertise in one RDS system instead of maintaining distributed data and fragmented knowledge. Strategy 1: Our initial plan - We began with a two-way AWS DMS setup from source to target and vice-versa. The idea was to migrate applications incrementally from MSSQL to Postgres. Why was this inefficient? 1. Two-way DMS had loopholes — loopback prevention wasn’t working efficiently in the latest version until the AWS team created a custom patch for us. 2. This approach required multiple code changes, increasing vulnerability to bugs. 3. It was time-consuming and costly, since DMS would need to run until all applications were migrated. 4. This meant dual database costs (MSSQL + Aurora), significant developer involvement, migration testing overhead, and more. That’s when we stopped and asked: “Is there a better, more efficient way?” And that’s when we found Babelfish. What is Babelfish? Babelfish provides a compatibility layer on top of PostgreSQL so the database can understand T-SQL (MSSQL) syntax. In short: Postgres engine + MSSQL understanding. What this means in practice: Babelfish exposes two ports: 1433 → MSSQL communication (TDS protocol) 5433 → PostgreSQL port (native Postgres) 1. No application changes required. 2. No long-running DMS replication. 3. Huge cost savings with Aurora Postgres, which is I/O-optimized. Applications can be gradually migrated to the Postgres port (5433) if needed—or you can continue using Babelfish indefinitely. This open-source technology helped us migrate 7 TB of data in just 2 months (including testing and validation) and save 40% in RDS infrastructure costs. More on the HOW of the migration in the upcoming posts. The team that made it possible with a lot of learnings : Vijay Ramachandran Raghav Bhutra Ajay Bandari MediBuddy

From On-Prem Complexity to Cloud-Native Scale. This diagram captures a journey many enterprises are on today modernizing from traditional data centers to a fully cloud-native AWS architecture. Before Cloud Migration: > Heavy reliance on corporate data centers. > Regionally replicated databases with operational overhead. > VPN-based connectivity for backups and hybrid workloads. > Limited scalability and slower innovation cycles. After Cloud Migration: > Fully managed AWS cloud services across regions (US, EU, APAC). > Scalable compute with Auto Scaling, Lambda, and managed databases. > Global performance via CloudFront, caching with ElastiCache. > Simplified data analytics using Athena and Redshift. > Secure access with SAML SSO (Okta). > Reduced ops burden, higher resilience, and faster time-to-market. Key Takeaway: Cloud migration isn’t just about “moving servers” it’s about rethinking architecture to unlock scalability, resilience, and innovation while reducing operational complexity. If you’re designing or migrating large-scale systems, investing time in the target architecture makes all the difference. #CloudMigration #AWS #CloudArchitecture #SolutionsArchitecture #DigitalTransformation #HybridCloud #CloudNative #DevOps #Scalability

I've ensured 100+ AWS migration projects succeed. Found key reasons why migrations could fail. (This is how we solved it, and you can too) 1. Ever-changing migration plans Constantly changing your migration plan, like 'Lift and Shift', 'Re-platforming', 'Re-hosting' etc., is a red flag. This inconsistency can lead to unforeseen dependencies and legacy system issues. To mitigate this, conduct thorough application dependency mapping and discovery before planning migration phases. 2. Inconsistent migration methods In a multi-tier web application migration project, using different methods like 'Re-hosting', 'Re-platforming', and 'Refactoring' for different applications will prove inefficient. It can lead you to integration issues and performance bottlenecks. Avoid it by proper standardization, defining clear target architectures, and grouping similar applications together. 3. Ineffective escalation process In a large data warehouse migration project, you can face issues with data consistency and integrity. These technical issues need to be promptly escalated to the right team for quick resolution. As a solution, establish a strict governance structure and communication plan to ensure blockers reach the right teams promptly. 4. Late emerging migration issues While doing CRM system migration, unforeseen data migration complexities can surface late, causing delays and significant rework. To address this, implement mechanisms like early design processes, tools, and escalation paths to identify issues sooner and maintain project momentum. 5. Lack of stakeholder alignment This can usually be faced while undergoing an ERP system migration. Stakeholder buy-in can prove to be critical. Without alignment, miscommunication between the migration team and business stakeholders can lead to roadblocks. Ensure alignment early by highlighting how AWS benefits specific objectives, fostering strong support throughout the migration process. Just remember that the future is unpredictable. But if planned well, then things are manageable! In the same way, Murat Yanar, Director at Amazon Web Services (AWS), once said, “You may not be able to predict the future needs of your business precisely. But the AWS cloud provides services to meet these ever-changing demands and help you innovate flexibly and securely.” Curious to know: What’s your biggest challenge when it comes to AWS migration? #aws #database #scalability #softwareengineering #simform

**My AWS Cloud Migration Project 🚀☁️ Simple & Secure Hybrid Design!** Ever wondered how to move a company from its own computers to the cloud safely and smoothly? 🤔 I'm sharing the plan I made for moving a dating app ("Lovely") to AWS, connecting it with their existing setup! It was my final project for the AWS Cloud Architect course at School of Hi-Tech and Cyber Security Bar-Ilan University. Here’s a peek at the main ideas: ✅ **Easy & Secure Logins:** Made it simple for users to log in safely using their existing work accounts (Azure AD) with extra security checks (MFA). Set up separate AWS areas for different teams like R&D, IT, and DevOps. ✅ **Watching the Money:** Kept track of spending with automatic alerts (AWS Budgets & CloudWatch) to avoid surprises. Managed all billing from one central spot (AWS Organizations & Control Tower). ✅ **Connecting Old & New:** Safely linked the company's offices to AWS using a secure connection (Site-to-Site VPN). Made sure some computers could reach the internet without being directly exposed (NAT gateways). ✅ **Keeping the App Running Smoothly:** Moved their WordPress website to flexible AWS computers (EC2), databases (RDS), and storage (EFS). Ensured the site stays up even if parts fail (Multi-AZ, Auto Scaling, ALB) and kept user data safe (HTTPS, KMS). ✅ **Smart & Safe Storage:** Used AWS S3 like digital filing cabinets, giving each team their own secure folder. Protected all files with secret codes (KMS) and set rules to save money and make backup copies elsewhere automatically. ✅ **Top-Notch Security:** Limited access to only approved locations (IP restrictions), used unique keys for computers (EC2 Key Pairs), and stored passwords securely (Secrets Manager). Ensured all data was scrambled (encrypted) when stored or sent. ✅ **Automation Power:** Created little helpers (Lambda & EventBridge) to automatically turn off unused computers, saving money. Kept a close eye on everything with monitoring tools (CloudWatch). ✅ **Ready for Anything:** Prepared a backup website in a different location just in case (Disaster Recovery). Automatically copied important data to another region (S3 Replication) for extra safety. **Tools / Tech Used** 💻🛠️ ☁️ AWS: EC2, RDS, EFS, S3, KMS, IAM, Organizations, Control Tower, Budgets, CloudWatch, Lambda, EventBridge, VPC, VPN, NAT Gateway, ALB, Route 53, Secrets Manager 🔑 Identity: Azure AD, SAML, MFA 🔒 Security: Fortinet 💻 Other: VMware, WordPress What do you think of this setup? Let me know your thoughts in the comments! 👇 Follow me for more cloud project insights! #AWS #CloudArchitecture #HybridCloud #SolutionArchitect #CloudSecurity #CloudMigration #DevOps #CyberSecurity #Project #Learning ---

I was talking with a friend about an upcoming cloud migration to Oracle, and the conversation quickly shifted from providers to something more important. The real question was not where to migrate, but how to migrate well. What I shared with her was a three-phase approach I have worked through in AWS migrations from on-prem. 1) Assess. This is where migrations succeed or fail before any workload ever moves. Cloud readiness assessments, a clear business case, and TCO modeling help align people, business, governance, platform, security, and operations. If you skip this step, everything downstream becomes reactive. 2) Mobilize. This is where foundations are built. A Cloud Center of Excellence, landing zones, connectivity, security baselines, and initial proof of concept applications. This phase is about learning fast and establishing patterns, not moving everything at once. 3) Migrate and Modernize. This is where the 7 Rs come into play. Rehost, replatform, relocate, refactor, retain, retire, and repurchase. Most environments are a mix, and the goal is not perfection, but intentional decisions based on time, cost, and complexity. What I appreciate about these conversations is the reminder that cloud migrations are rarely about technology alone. They are about clarity, sequencing, and shared understanding across teams. If your company is planning a migration or modernization effort next year, I suggest that you start with the whiteboard before the console to map out ideas. Here is the image from our whiteboarding session, along with another I created using Nano Banana to bring it to life. Curious how others approach the Assess and Mobilize phases in their migrations. What lessons have you learned early that saved you later?