Data Analysis and Decision-Making

🚨The greatest drop-off is from Product Details Page To Cart Page, so we must improve our Product Details Page! Not so fast ✋ In today's age of data obsession, almost every company has an analytics infrastructure that pumps out a tonne of numbers. But rarely do teams invest time, discipline & curiosity to interpret numbers meaningfully. I will illustrate with an example. Let's take a simple e-commerce funnel. Home Page ~ 100 users List Page ~ 90 users Product Display Page ~ 70 users Cart Page ~ 20 users Address Page ~ 15 users Payments Page ~12 users Order Confirmation Page ~ 9 users A team that just "looks" at data will immediately conclude that the drop-off is most steep between Product Details Page & Cart Page. As a consequence they will start putting in a lot of fire power into solving user problems on Product Display Page. But if the team were data "curious", would frame hypothesis such as "do certain types of users reach cart page more effectively than others?" and go on to look at users by purchase buckets, geography, category etc and look at the entire funnel end to end to observe patterns. In the above scenario, it's likely that the 20 cart users were power users whilst new & early purchasers don't make it to this stage. The reason could be poor recommendations on the list page or customers are only visiting the product display page to see a larger close up of the product. So how should one go about looking at data ? Do ✅ Start with an open & curious mind ✅ Start with hypothesis ✅ Identify metrics & counter metrics that will help prove/disprove hypothesis ✅ Identify the various dimensions that could influence behaviours - user type, geography, category, device type, gender, price point, day, time etc. The dimensions will be specific to your line of business. ✅ Check for data quality and consistency ✅ Look at upstream and downstream behaviour to see how the behaviour is influenced upstream and what happens to the behaviour downstream. ✅ Check for historical evidence of causality Dont ❌ Look at data to satisfy your bias ❌ Rush to conclude your interpretation ❌ Look at data in isolation - - - TLDR - Be curious. Not confirmed. #metrics #analytics #productmanagement #productmanager #productcraft #deepdiveswithdsk

This article maps bias across the full lifecycle of medical AI: training data (who is in the dataset and what’s missing) --> labels (how “ground truth” encodes human bias) --> model development and evaluation --> real-world implementation --> which models get published and from where. It illustrates concrete clinical risks, from melanoma models that underperform on dark skin to ICU mortality models with recall as low as 25% in underrepresented groups, and shows how biased systems can drive substandard decisions for the very patients who most need better care. The authors argue that mitigation must go beyond technical fixes, combining diverse datasets, fairness-aware modeling, interpretability, stronger standards, and clinical trials that explicitly test for unbiased performance. Key takeaways - Bias enters early: imbalanced cohorts, nonrandom missing data, and the absence of social determinants of health all push models to work best for already advantaged groups. - “Ground truth” is not neutral: labels reflect provider behavior, misclassification, and structural inequities, so models can learn and amplify existing clinical biases rather than correct them. - Whole-cohort metrics like AUC can hide harm; subgroup performance, fairness metrics, and interpretability tools are essential to detect and mitigate inequity in model outputs. - Real-world deployment introduces new bias: models can fail on populations unlike the training data (Epic sepsis model is a key example), and clinician use/override patterns can themselves be inequitable. - Publication and funding ecosystems skew what gets built and validated, with over half of clinical AI models using US or Chinese data, and radiology dominating the literature. Dipu’s Take If AI in medicine isn’t explicitly designed and governed for equity, it will quietly operationalize our worst blind spots at scale. Accuracy alone is a distraction metric; the harder questions are “for whom, in which contexts, and at what clinical cost?” The leadership opportunity here is to treat debiasing as core safety and quality work: mandate diverse data, require subgroup reporting and fairness metrics, bake bias monitoring into post-deployment oversight, and tie reimbursement and approvals to demonstrated equitable performance in trials.

We don’t run our businesses on lagging indicators. Why are we still managing pregnancy this way? Saturday I had the chance to talk about the intersection of predictive intelligence, wearables, and care delivery with Elizabeth Cherot MD, MBA and Stephanie Rouse from Lucina and Unified Women's Healthcare and Chris Curry from ŌURA. A few takeaways I’m still noodling: 1. Data is only valuable if it changes what happens next. Healthcare has no shortage of dashboards and “insights.” What we need is actionability. If risk is identified but no one intervenes, it’s trivia, not transformation. The opportunity with data is no longer just understanding what’s happening; it’s using that understanding to change what happens next. At Lucina, we use health data to build explainable models so the next best action of care is clear. 2. AI + human care is where magic lives. The best use of technology isn’t removing humans from care. It’s helping the right people show up at the right time with the right information to drive better patient outcomes and better efficiency. 3. The future of maternity care is proactive and real time. Too much of women’s healthcare still waits for a problem to reveal itself. Smart models identify risk earlier, hyper-personalize recommendations, so care teams can act before complications escalate. While claims and EHR data are comprehensive, wearable data gives real-time insights before the visit or before claims drop. 4. Wearables are powerful shared decision-making tools. Continuous data is everywhere and consumers dig it (me, I am consumers 👋 ). When integrated and used appropriately, it may help clinicians understand what their patients are living between visits. This creates better context for shared decision-making. When we can combine the right data with the right care model, we make care more proactive, more personalized, and ultimately more impactful. #WomensHealth #Innovation #AI #Data

In data analytics, you must focus on delivering clear insights, not complexity. Here are steps you can use to find a valuable but simple solution: 1. 𝗨𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱 𝘁𝗵𝗲 𝗰𝗼𝗿𝗲 𝗯𝘂𝘀𝗶𝗻𝗲𝘀𝘀 𝗾𝘂𝗲𝘀𝘁𝗶𝗼𝗻: What does the stakeholder need to know? What are the questions they need to answer? Which decisions do they have to make?   2. 𝗖𝗵𝗲𝗰𝗸 𝘁𝗵𝗲 𝘂𝘀𝗲 𝗰𝗮𝘀𝗲: Determine if it’s a one-time analysis or a solution that needs regular updates. Will it be a stand-alone solution or does it need to be integrated with other products?   3. 𝗨𝘀𝗲 𝗳𝗮𝗺𝗶𝗹𝗶𝗮𝗿 𝘁𝗼𝗼𝗹𝘀: Stick with SQL and Excel when they can do the job. Only introduce new tools if they add a significant additional value.   4. 𝗙𝗼𝗰𝘂𝘀 𝗼𝗻 𝗮𝗰𝘁𝗶𝗼𝗻𝗮𝗯𝗹𝗲 𝗶𝗻𝘀𝗶𝗴𝗵𝘁𝘀: Deliver what stakeholders can directly use to make decisions to steer the business.   5. 𝗨𝘀𝗲 𝘀𝗶𝗺𝗽𝗹𝗲 𝘃𝗶𝘀𝘂𝗮𝗹𝗶𝘇𝗮𝘁𝗶𝗼𝗻𝘀: Most of the time simple bar and line charts will do the job and deliver the same or better insights than complex visualizations.   6. 𝗜𝘁𝗲𝗿𝗮𝘁𝗲 𝗯𝗮𝘀𝗲𝗱 𝗼𝗻 𝗳𝗲𝗲𝗱𝗯𝗮𝗰𝗸: Start simple and build complexity only if necessary. Change is the only constant in most data projects.   7. 𝗞𝗲𝗲𝗽 𝘁𝗵𝗲 𝗣𝗮𝗿𝗲𝘁𝗼 𝗽𝗿𝗶𝗻𝗰𝗶𝗽𝗹𝗲 𝗶𝗻 𝗺𝗶𝗻𝗱: Often you will get 80% of the results with 20% of the effort. Check if that's good enough for the business and only go the extra mile when necessary. Keep your solutions simple, and deliver what truly matters for the business! What’s your strategy to avoid building overcomplicated solutions? ---------------- ♻️ 𝗦𝗵𝗮𝗿𝗲 if you find we should focus more on value than complexity ➕ 𝗙𝗼𝗹𝗹𝗼𝘄 for more daily insights on how to grow your career in the data field #dataanalytics #datascience #keepitsimple #productivity #careergrowth

What if one of the most important medical decisions of your life came down to five rushed minutes, and incomplete data? In a recent conversation with Fidji Simo, CEO of Applications at OpenAI, she shared a moment that should give every healthcare leader, operator, and technologist pause. While hospitalized, she was about to be given a standard antibiotic for a routine infection. On the surface, it was the correct protocol. But by quickly cross-referencing the drug against her full medical history using AI, she uncovered a critical risk. It could have reactivated a serious past C. diff infection. The physician’s response was telling. “I have five minutes to make rounds. I can’t review years of records.” Modern healthcare is still operating on fragmented data, siloed specialties, and time-constrained decision-making. Even the best clinicians are forced to make high-stakes calls without full context. And this is where the opportunity becomes clear. We are entering a new era where AI is not replacing clinicians, but augmenting their ability to see the whole picture. By connecting longitudinal health data, labs, genomics, wearables, and medical history, we can move from reactive care to truly informed, real-time decision-making. In our full discussion, we explore what this shift means at scale: • Why most clinical errors are not about knowledge gaps, but missing context • How fragmented health systems create unnecessary risk and inefficiency • What it looks like when AI becomes a layer of intelligence across the entire patient journey • So much more This is a systems design problem, a data problem, and ultimately, a leadership problem. The organizations that solve for context, not just care delivery, will define the future of health. Listen to our full conversation here: https://lnkd.in/g_2FsR2q

How often have you seen developers or agencies touting your desktop PageSpeed score as a measure of success? How frequently have you observed them misleading with these lab scores? As an online brand, it's crucial to get this right. Don't settle for less—ask the right questions and track meaningful metrics. The only metrics that matter for your online growth regarding website speed are Real User Monitoring (also known as Field Data or #CoreWebVitals). Chasing a lab score like 90/100 is unnecessary because your performance may appear excellent, but your website could still fail the Core Web Vitals assessment. Next time your agency or developer says your #PageSpeed is poor, ask: ▶ What data are you referring to? Lab or Field? ▶ If they say "Lab," you know you're not in the right hands. ▶ If they say "Core Web Vitals" and mention needing 90/100, again, you know you're not in the right hands. Core Web Vitals can’t be tracked in a lab environment; they are meant for real users in real-time. To track these, you need Real User Monitoring solutions. You have two options: ▶ Create your own RUM ▶ Use RUM SaaS tools (like Auditzy™ RUM) Now you must be wondering, Does lab data really make no sense? It does, but only for developers to understand code pain points under a simulated environment, provided they are looking at a comprehensive lab report based on multiple test profiles. Relying solely on PSI for your lab report will lead to difficulties. The correct way to measure lab data: ▶ Use tools where you can select devices, network speeds, and test server locations. ▶ Create multiple test profiles (reflecting your website visitor personas) and analyze how lab scores shape up. You can use synthetic monitoring tools like Auditzy™ Synthetic to achieve this. Summary: ▶ Only Core Web Vitals matter for Google. ▶ Lab data is not the right metric for tracking website performance. P.S. In the image below, the desktop score is 90, yet the desktop Core Web Vitals assessment fails. Can you identify why?

🔄 Demystifying Data Processing Architectures. 🧠 Ever wondered how data flows from raw logs to real-time insights? Whether you're just starting out in data engineering or leading architecture decisions—understanding the spectrum of data processing models is your edge. From batch jobs that crunch data overnight to real-time systems that react in milliseconds—choosing the right architecture isn't just technical, it's strategic. Here's a visual breakdown of the 6 major paradigms: 🔹 𝗕𝗔𝗧𝗖𝗛 𝗣𝗥𝗢𝗖𝗘𝗦𝗦𝗜𝗡𝗚  • Latency: Hours-Days | Cost: Low | Accuracy: Highest  • Perfect for: Historical analysis, compliance reporting  • Tech: Spark, MapReduce, SQL ETL 🔹 𝗠𝗜𝗖𝗥𝗢-𝗕𝗔𝗧𝗖𝗛  • Latency: Seconds-Minutes | Cost: Medium | Accuracy: High  • Perfect for: Real-time dashboards, trend analysis  • Tech: Spark Streaming, Storm Trident 🔹 𝗡𝗘𝗔𝗥 𝗥𝗘𝗔𝗟-𝗧𝗜𝗠𝗘  • Latency: Sub-second to Minutes | Cost: Medium-High  • Perfect for: Operational monitoring, business alerts  • Tech: Kafka, Complex Event Processing 🔹 𝗦𝗧𝗥𝗘𝗔𝗠 𝗣𝗥𝗢𝗖𝗘𝗦𝗦𝗜𝗡𝗚  • Latency: Milliseconds | Cost: High | Accuracy: Good  • Perfect for: Fraud detection, live personalization  • Tech: Apache Flink, Kafka Streams Each has its own sweet spot—whether you're building dashboards, detecting fraud, or automating decisions. How to Decide? ✔️ High accuracy, huge data, non-urgent → Batch ✔️ Need live dashboards, but tolerable delay → Micro-Batch/Real-Time ✔️ Instant actions (fraud, alerts, in-game events) → Stream 💬 Curious how your system stacks up? Want to choose the right model for your next project? 👉 Dive into this visual guide and comment below: Which architecture are you using today—and why? Let’s spark a conversation that bridges learning and leadership in data engineering. Stay tuned for more such Data Engineering concepts with Pooja Jain! #Data #Engineering #BigData #Analytics

Want to up your data analysis/science game? I will share one of my most powerful techniques in this post. Here’s the cool part. This technique is universal. I’ve used it to feed exploratory data analysis (EDA), market basket analysis, and machine learning algorithms. I’ve used it with small data, big data, and everything in between. Can you guess what it is? It’s a specific type of SQL query: SELECT <unique_id> -- Has something happened? ,MAX(CASE WHEN <some_logical_condition> THEN 1 ELSE 0 END) AS <indicator> -- Count how many times something happened ,SUM(CASE WHEN <some_logical_condition> THEN 1 ELSE 0 END) AS <count> -- Count how many times something happened within X number of days ,SUM(CASE WHEN <some_logical_condition>  AND DATEDIFF(DAY, <start_date>, <end_date>) <= <value> THEN 1 ELSE 0 END) AS <date_count> FROM <some_table>     <any_joins> WHERE <filter> GROUP BY <unique_id> I can’t tell you how often I’ve used some version of the above SQL to craft data that produced new business insights. Some real-world examples: 1 - Pull data into Microsoft Excel (e.g., via Power Query) to conduct EDA. 2 - Crafting binary indicators to use in market basket analysis. 3 - Building powerful features for machine learning models. Over the years, I’ve found SQL to be the most versatile and useful of all my data skills: A – Querying relational databases for “small” data. Make no mistake, “small” relational data is still king in many organizations. B – Querying “big data” stores like Spark and Hive. That said, the idea behind the SQL query is the real magic. Grab your tool of choice and start exploring your data: You can reproduce the SQL using dplyr or pandas? Awesome! You can reproduce the SQL using a drag-and-drop visual tool? Sweet! You can reproduce the SQL using M/DAX/VBA in Excel? Righteous! I’m betting you won’t be disappointed: BTW – I’ve consistently found that <date_count> features are the most useful for uncovering new business insights, especially with machine learning models. Stay healthy and happy data sleuthing! #datascience #machinelearning #analytics #businessanalytics #dataanalytics

How I saved a tech company $128,557 in 22 days (without hiring more staff): $128,557. Gone. All because they didn’t trust their CRM to do its job. Here’s the story: - 185 employees - A growing tech company - Held back by CRM inefficiencies Their marketing director was wasting 5.5 hours a day on low-level tasks: Manually entering lead data into three CRMs: - HubSpot - Google Sheets - Salesforce The result? - Typos and bad data - Gut-feeling decisions - Delayed reporting Let’s crunch the numbers: 110 hours a month—lost. 165 working days a year—wasted. A staggering $128,557—down the drain. Here’s how we fixed it: First, discovery. We identified every inefficiency and bottleneck. Then, build. We integrated HubSpot with Salesforce and eliminated Google Sheets. Finally, testing. Everything was run in Salesforce Sandbox, approved, and launched. 22 days later, their system ran like clockwork. The result? 110 hours saved every month $128,557 in yearly costs eliminated And their marketing director? Back to focusing on real high-level work. P.S. What’s one inefficiency that’s holding your business back right now?

Linking health data to location data sounds straightforward. It took years of specialist work to make it possible without compromising either the data or the people behind it. We were brought in to work on one of the most ambitious data integration programmes in the UK public sector. The platform was designed to help researchers and analysts discover, join, and analyse data. Previously, that data existed in separate silos across government departments. The challenge was not a shortage of data. The UK holds extraordinary datasets covering health, labour markets, demographics, and geography. The challenge was that each dataset had been built with different definitions, geographies, and privacy requirements. Linking them without careful architecture risked exposing personal information. It also produced analysis that was fundamentally unreliable. Neither was acceptable. Here's what we delivered. We built privacy-preserving anonymisation workflows for every dataset ingested into the platform. Each workflow included differential risk controls and automated disclosure checks. Not as a compliance layer applied afterwards. As a core architectural component built into the ingestion process from the start. We implemented a reference data hub that unified geospatial codes, health lookups, labour market data, and demographic classifications. Everything was brought into a single governed catalogue. This solved a problem that had prevented meaningful cross dataset analysis for years. Every dataset now carries a common location spine. This allowed health outcomes to be examined alongside labour market data and census boundaries. The analysis could be performed using consistent geographies that did not drift between sources. We built APIs enabling analysts to combine datasets in ways that were previously manual, error prone, and slow. The platform was designed to scale to billions of records as participation from additional departments grows. The outcomes. Researchers can now discover and analyse previously siloed data to accelerate evidence based policy design. Robust anonymisation and governance frameworks reduced the risks associated with data sharing. As a result, departments that previously held back are now participating. Geospatial alignment means every analysis carries consistent national and regional context rather than fragmentary local snapshots. The hardest data problems are rarely about storage or processing power. They are about the invisible barriers between datasets. Different codings, different boundary definitions, different privacy thresholds. Building the infrastructure that lets disparate data speak a common language is painstaking, specialist work. But it is what transforms individual datasets into genuine analytical capability. What siloed data in your organisation could generate transformative insight if it could reliably connect to other sources? #DataIntegration #PrivacyPreserving #PublicSector