Privacy breaches have become increasingly prevalent, exposing individuals to significant risks. These breaches can have far-reaching consequences, including identity theft and life-threatening situations. Several studies have analysed data and privacy breaches and presented detection or prevention techniques to combat these breaches. However, because the number and type of breaches have significantly increased, these studies have become less relevant or outdated. Previous research on data/privacy breaches compared the techniques and results of various studies, but none attempted to comprehensively analyse the type of information and the type and level of compromise that occurred after such breaches. In this survey, we examine the fundamental concepts of privacy and security and define the security incidents and data/privacy breaches. We propose a set of criteria to evaluate the published studies on privacy breaches. We thoroughly investigate the problem domains and security-related concerns considering six recent breach cases in Australia, elucidating the critical challenges and issues associated with privacy breaches. We comprehensively review and outline the trends of security incidents and data/privacy breaches from 2020 to 2023. Additionally, we review the current state-of-the-art countermeasures to safeguard against these breaches. Finally, we identify an open research direction to develop an artificial intelligence (AI)-powered security framework that can help analyse cyber threats, characterise attackers' behaviours, distinguish between legitimate and illegitimate privacy policies, and restrict access to individuals' information. Overall, this survey will help organisations to reassess and update their security and privacy measures.