The use of electronic surveillance by the United Kingdom grew from the development of signal intelligence and pioneering code breaking during World War II. In the post-war period, the Government Communications Headquarters (GCHQ) was formed and participated in programmes such as the Five Eyes collaboration of English-speaking nations. This focused on intercepting electronic communications, with substantial increases in surveillance capabilities over time. A series of media reports in 2013 revealed bulk collection and surveillance capabilities, including collection and sharing collaborations between GCHQ and the United States' National Security Agency. These were commonly described by the media and civil liberties groups as mass surveillance. Similar capabilities exist in other countries, including western European countries.

Surveillance of electronic communications in the United Kingdom is regulated by acts of Parliament. In particular, access to the content of private messages (that is, interception of a communication such as an email or telephone call) must be authorised by a warrant signed by a Secretary of State. Although the law provides for governance and safeguards over the use of electronic surveillance, these safeguards have been criticised as not far-reaching enough, nor protective enough of the public's privacy; Further oversight including a requirement for judges to review warrants authorised by a Secretary of State, as well as new surveillance powers, were introduced by the Investigatory Powers Act 2016.

The judicial body which oversees the intelligence services in the United Kingdom, the Investigatory Powers Tribunal, ruled in December 2014 that the legislative framework in the United Kingdom does not permit mass surveillance and that while GCHQ collects and analyses data in bulk, its practices do not constitute mass surveillance. Other independent reports, including one by the Intelligence and Security Committee of Parliament, also came to this view although they found past shortcomings in oversight and disclosure, and said the legal framework should be simplified to improve transparency. However, notable civil liberties groups and broadsheet newspapers continue to express strong views to the contrary, while UK and US intelligence agencies and others; have criticised these viewpoints in turn.

Various government bodies maintain databases about citizens and residents of the United Kingdom. These include "bulk data sets" such as medical records. In January 2016 the Home Secretary stated she would neither restrict the data sets that might be accessed for such purposes, nor state whether or not communications protected from law enforcement access such as journalist's sources and legal privilege had been accessed covertly. Although the use of video surveillance cameras in the United Kingdom is common, as it is in many countries, its prevalence may historically have been overstated. Legal provisions exist that control and restrict the collection, storage, retention, and use of information in government databases, and require local governments or police forces operating video surveillance cameras to comply with a code of conduct: the Surveillance Camera Code of Practice.

Legal framework for lawful interception

The legal framework in the United Kingdom for lawful interception and storage of communications data and, when a warrant exists, the content of electronic communications is based on the Regulation of Investigatory Powers Act 2000 and several other pieces of legislation. The Data Retention and Investigatory Powers Act (DRIPA) 2014 deals with the retention of certain types of communications data (not the content of messages). It was brought into effect after the European Union's Data Retention Directive was declared invalid. The Telecommunications Act 1984 has also been used by the government to facilitate bulk communications data collection. The Protection of Freedoms Act 2012 includes several provisions related to controlling or restricting the collection, storage, retention, and use of information in government databases. The Human Rights Act 1998 requires the intelligence agencies, including GCHQ, to respect citizens' rights as enumerated in the European Convention on Human Rights.

Compatibility with human rights law The Investigatory Powers Tribunal ruled in December 2014 that the legal frameworks in the United Kingdom governing the bulk interception of data and intelligence sharing with agencies in other countries do not breach the European Convention on Human Rights, and are compliant with Articles 8 (right to privacy) and 10 (freedom of expression) of the European Convention on Human Rights. However, the Tribunal stated that one particular aspect of intelligence sharing, the data-sharing arrangement that allowed UK Intelligence services to request data from the US surveillance programmes Prism and Upstream, had been in contravention of human rights law until two paragraphs of additional information, providing details about the procedures and safeguards, were disclosed to the public in December 2014.

Privacy and civil liberties advocates such as Liberty and Privacy International, who brought a legal case against the government to force the judgement, continue to oppose to the temporary bulk collection of data, powers to access this and retain selected data, as well as intelligence sharing relationships; they intend to appeal the judgement to the European Court of Human Rights. Intelligence agencies and MPs have criticised the viewpoint of privacy campaigners on this issue.

Following the publication of a special report by the Intelligence and Security Committee of Parliament in March 2015, which identified shortcomings in past oversight and potential improvements to the legislative framework, Prime Minister David Cameron initiated an inquiry into the legalisation governing the interception powers of the intelligence agencies.

A third independent report into surveillance in the UK published in July 2015 found the intelligence agencies are not knowingly carrying out illegal mass surveillance of British citizens. However, it did say the laws governing the agencies' powers to intercept private communications need a significant overhaul. This view is consistent with separate reports by the Interception of Communications Commissioner.;

In October 2016, the Investigatory Powers Tribunal ruled British security services had, in secret, unlawfully collected citizens' information, including financial information, individual phone and web use, and other confidential personal data, without adequate safeguards or supervision for 17 years. The tribunal found that from its inception in 1998 until its public acknowledgement on 4 November 2015, this bulk collection was in breach of article 8 of the European convention on human rights.

After a High Court victory by two MPs challenging the legality of DRIPA, the UK government appealed to the European Court of Justice (ECJ), but in December 2016 the verdict of the national court was upheld. The ECJ ruled that general and indiscriminate retention of emails and electronic communications by governments was illegal, opening the way to challenges against the UK's new Investigatory Powers Act (2016), which replaced DRIPA.

Investigatory Powers Act 2016

, the Investigatory Powers Bill was published for scrutiny. The bill would introduce new powers, as well as restate existing ones, for targeted interception of communications, bulk collection of communications data, and bulk interception of communications. New oversight procedures would be introduced, including a requirement for a judge to review a warrant signed by a Minister for interception of communications, that is reading the content of messages. The bill would require Internet connection records – which websites were visited but not the particular pages and not the full browsing history – to be kept by internet service providers for one year.

On 16 November 2016, the Investigatory Powers Bill had passed both houses of parliament and is scheduled to become law.

The Investigatory Powers Act 2016 is a comprehensive statute which makes provision for both targeted and bulk retention of content and metadata. It consolidates much of the previous legislation and makes public a number of previously secret powers (equipment interference, bulk retention of metadata, intelligence agency use of bulk personal datasets).

Controversially, it enables the Government to require internet service providers and mobile phone companies to maintain records of (but not the content of) customers' Internet connections for up to 12 months. Police and intelligence officers may seek approval for access to these records without a warrant, as part of a targeted investigation.

In addition, the Act creates new safeguards, including a requirement for judges to approve the warrants authorised by a Secretary of State before they come into force.

The Bill that gave rise to the Act was informed by reports of seven parliamentary committees, as well as an external report from the Royal United Services Institute and two influential reports by David Anderson QC, the UK's Independent Reviewer of Terrorism Legislation: A Question of Trust (2015) and the report of his Bulk Powers Review (2016). The latter contains a detailed appraisal (with 60 case studies) of the operational case for the bulk powers used by MI5, MI6 and GCHQ and often characterised as mass surveillance.

The Act may yet require amendment as a consequence of legal cases brought before the Court of Justice of the European Union and the European Court of Human Rights.

Data Retention and Investigatory Powers Act 2014

In April 2014, the European Court of Justice ruled that the European Union's Data Retention Directive was invalid. Since October 2007, telecommunication companies had been required to keep records of phone calls and text messages for a minimum of 6 months and at most 24 months under this directive. The European Court of Justice found it violates two basic rights, respect for private life and protection of personal data.

Supported by all three major political parties in the UK, Parliament passed the Data Retention and Investigatory Powers Act in July 2014 to ensure the police and security services retained their existing powers to access telephone and internet records. No additional powers were granted by the legislation, but it did make clear that the requirements also apply to foreign companies, based abroad, whose telephone and internet services are used in the UK.

The data being retained does not include the content of messages and telephone calls, just metadata describing when and who the users contacted by email, telephone, or text message. In circumstances when the Home Secretary issues a warrant for intercepting the content of private messages, the Act clarifies the law with which internet services providers must comply.

Provisions were included in the Act to "increase transparency and oversight"; the BBC reported that this included the following: The creation of a Privacy and Civil Liberties Oversight Board to scrutinise the impact of the law on privacy and civil liberties. Annual government transparency reports on how these powers are used. The appointment of a senior former diplomat to lead discussions with the US government and internet firms to establish a new international agreement for sharing data between legal jurisdictions. A restriction on the number of public bodies, including Royal Mail, able to ask for communications data under the Regulation of Investigatory Powers Act (RIPA). Inclusion of a termination clause ensuring the powers under this Act expire at the end of 2016. Statement that a wider review of the powers needed by government should be undertaken during the next parliament (after the general election in May 2015).