Data, data and then some more data

data can be the difference between an AI solution that seems to work and one that does.

data can be the reason your ML model doesn't work well:

"the healthcare fraud problem is an inherently imbalanced problem with a small number of known fraudulent providers and many non-fraudulent providers. This causes machine learning algorithms to become biased towards the majority class, and many times, causes the learner to have low classification performance on the fraudulent class." ¹

Figure: Phases of the CRISP-DM reference model ²

diverse data helps generalize well during language model pretraining.

I curated a mixed domain dataset to train an SLM so that it generalized well and my model's perplexity improved by 15% as compared to using a single-domain dataset.

data can be used to make design decisions for your AI system.

I observed that the desired ground truth ("golden") data for my RAG Q&A system rewarded precision over recall i.e. including only the minimally sufficient chunks so I introduced a precision-targeted filtering funnel for choosing chunks. That improved precision by 15% while trading off recall by 3%.

data can be used to determine failure modes of your system. and then help design for them.

I figured out that complex multi-part questions were not being answered completely by my fine-tuned model (for the downstream task of schema linking) and so I fine-tuned a specialized adapter to handle only question decomposition before answering the question.

the lesson I learnt was when building an AI system to fulfil a downstream task, make data diverse, measure the relevant metrics, diagnose which edge cases break performance and then do this repeatedly until you reach good performance. make data-driven decisions, don't opt in for coolness or complexity of the system.

bonus tip: a good way to do this is to deploy your system and use real user data for detecting poor performance edge cases (with appropriate consent, of course).

this blog is the first of my data-centric AI series just to give you a taste of how important it is to focus on data - from adding diversity to making data-driven decisions. see you in the next one where I share how I curated datasets!

If you enjoyed this, I'm glad. I'm here to talk about real AI-powered projects from experience. from building, failing and learning so you can start from a stronger footing. always happy to hear from readers and fellow Applied AI builders! let me know what resonated with you!

References

¹Johnson JM, Khoshgoftaar TM. Data-Centric AI for Healthcare Fraud Detection. SN Comput Sci. 2023;4(4):389. doi: 10.1007/s42979-023-01809-x. Epub 2023 May 11. PMID: 37200563; PMCID: PMC10173919.

² Chapman P, Clinton J, Kerber R, Khabaza T, Reinartz T, Shearer C, Wirth R. CRISP-DM 1.0: step-by-step data mining guide. CRISP-DM consortium: NCR Systems Engineering Copenhagen (USA and Denmark) DaimlerChrysler AG (Germany), SPSS Inc. (USA) and OHRA Verzekeringen en Bank Groep B.V (The Netherlands), 2000; CRISP-DM consortium: NCR Systems Engineering Copenhagen (USA and Denmark) DaimlerChrysler AG (Germany), SPSS Inc. (USA) and OHRA Verzekeringen en Bank Groep B.V (The Netherlands).