# Data, data and then some more data

data can be the difference between an AI solution that seems to work and one that does.

**data can be the reason your ML model doesn't work well:**

"*the healthcare fraud problem is an inherently imbalanced problem with a small number of known fraudulent providers and many non-fraudulent providers. This causes machine learning algorithms to become biased towards the majority class, and many times, causes the learner to have low classification performance on the fraudulent class.*" <sup>1</sup>

![Phases of the CRISP-DM reference model](https://cdn.hashnode.com/uploads/covers/6ae7c2a5b98ee2dd74921a90/f7052aee-15ae-4d05-96d5-230dc79d28a1.png align="center")

**Figure: Phases of the CRISP-DM reference model <sup>2</sup>**

**diverse data helps generalize well during language model pretraining.**

I curated a mixed domain dataset to train an [SLM](https://simran-malik.github.io/projects/slm-pretraining/) so that it generalized well and my model's perplexity improved by 15% as compared to using a single-domain dataset.

**data can be used to make design decisions for your AI system.**

I observed that the desired ground truth ("golden") data for my [RAG Q&A system](https://simran-malik.github.io/projects/rag-chatbot/) rewarded precision over recall i.e. including only the minimally sufficient chunks so I introduced a precision-targeted filtering funnel for choosing chunks. That improved precision by 15% while trading off recall by 3%.

**data can be used to determine failure modes of your system. and then help design for them.**

I figured out that complex multi-part questions were not being answered completely by my [fine-tuned model (for the downstream task of schema li](https://simran-malik.github.io/projects/schema-linking/)nking) and so I fine-tuned a specialized adapter to handle only question decomposition before answering the question.

the lesson I learnt was when building an AI system to fulfil a downstream task, **make data diverse, measure the relevant metrics, diagnose which edge cases break performance and then do this repeatedly until you reach good performance. make data-driven decisions, don't opt in for coolness or complexity of the system.**

> **bonus tip:** a good way to do this is to deploy your system and use real user data for detecting poor performance edge cases (with appropriate consent, of course).

***this blog is the first of my data-centric AI series*** *just* *to give you a taste of how important it is to focus on data - from adding diversity to making data-driven decisions. see you in the next one where I share how I curated datasets!*

*If you enjoyed this, I'm glad. I'm here to talk about real AI-powered projects from experience. from building, failing and learning so you can start from a stronger footing. always happy to hear from readers and fellow Applied AI builders! let me know what resonated with you!*

### References

<sup>1</sup>Johnson JM, Khoshgoftaar TM. Data-Centric AI for Healthcare Fraud Detection. SN Comput Sci. 2023;4(4):389. doi: 10.1007/s42979-023-01809-x. Epub 2023 May 11. PMID: 37200563; PMCID: PMC10173919.

<sup>2</sup> Chapman P, Clinton J, Kerber R, Khabaza T, Reinartz T, Shearer C, Wirth R. CRISP-DM 1.0: step-by-step data mining guide. CRISP-DM consortium: NCR Systems Engineering Copenhagen (USA and Denmark) DaimlerChrysler AG (Germany), SPSS Inc. (USA) and OHRA Verzekeringen en Bank Groep B.V (The Netherlands), 2000; CRISP-DM consortium: NCR Systems Engineering Copenhagen (USA and Denmark) DaimlerChrysler AG (Germany), SPSS Inc. (USA) and OHRA Verzekeringen en Bank Groep B.V (The Netherlands).