Thursday, May 28, 2015

New details about the joint NSA-BND operation Eikonal

(Updated: June 30, 2015)

This weblog first reported about the joint NSA-BND operation Eikonal on October 15, 2014, but meanwhile interesting new details became available from the hearings of the German parliamentary inquiry, and from recent disclosures by a politician from Austria.

Under operation Eikonal, the NSA cooperated with the German foreign intelligence service BND for access to transit cables from Deutsche Telekom in Frankfurt. Here follows an overview of what is known about this operation so far. New information may be added as it comes available.





 

Initial reporting

Operation Eikonal was revealed by the regional German paper Süddeutsche Zeitung and the regional broadcasters NDR and WDR on October 4, 2014. They reported that between 2004 and 2008, the German foreign intelligence service BND had tapped into the Frankfurt internet exchange DE-CIX and shared the intercepted data with the NSA.

For this operation, NSA provided sophisticated interception equipment, which the Germans didn't had but were eager to use. Interception of telephone traffic started in 2004, internet data were captured since 2005. Reportedly, NSA was especially interested in communications from Russia.

To prevent communications of German citizens being passed on to NSA, BND installed a special program (called DAFIS) to filter these out. But according to the reporting, this filter didn't work properly from the beginning. An initial test in 2003 showed the BND that 5% of the data of German citizens could not be filtered out, which was considered a violation of the constitution.

Süddeutsche Zeitung reported that it was Deutsche Telekom AG (DTAG) that provided BND the access to the Frankfurt internet exchange, and in return was paid 6000,- euro a month. But as some people noticed, Deutsche Telekom was not connected to DE-CIX when operation Eikonal took place, so something didn't add up.

As we will see, this was right, and the actual cable tap was not at DE-CIX, but took place at Deutsche Telekom. Nonetheless, many press reports still link Eikonal to the DE-CIX internet exchange.



Operations center room in the former BND headquarters in Pullach
(Photo: Martin Schlüter - Click to enlarge)


Eikonal as part of RAMPART-A

As was first reported by this weblog on October 15, 2014, operation Eikonal was part of the NSA umbrella program RAMPART-A, under which the Americans cooperate with 3rd Party countries who "provide access to cables and host U.S. equipment".

Details about the RAMPART-A program itself had already been revealed by the Danish newspaper Information in collaboration with The Intercept on June 19, 2014. The program reportedly involved at least five countries, but so far only Germany and, most likely, Denmark have been identified.

On October 20, Information published about a document from NSA's Special Source Operations (SSO) division, which confirms that an operation codenamed "EIKANOL" was part of RAMPART-A and says it was decommissioned in June 2008.

The slide below shows that under RAMPART-A a partner country taps an international cable at an access point (A) and then forwards the data to a joint processing center (B). Equipment provided by the NSA processes the data and analysts from the host country can then analyse the intercepted data (C), while they are also forwarded to NSA sites in the US (D, E):




 

Parliamentary hearings

Because of the confusion about the role of Deutsche Telekom in operation Eikonal, the NSA investigation commission of the German parliament (NSAUA) decided to also investigate whether this company assisted BND in tapping the Frankfurt internet exchange.

During hearings of BND officials it became clear that operation Eikonal was not about tapping into the Frankfurt internet exchange DE-CIX, but about one or more cables from Deutsche Telekom. This was first confirmed by German media on December 4, 2014.


Hearing of November 6, 2014 (Live-blog)

According to witness T.B., who was heard on on November 6, 2014, it was just during the test period that the filter system was only able to filter out 95% of German communications. When the system went live, this percentage rose to 99% with a second stage that could filter out even more than 99%. When necessary, a final check was conducted by hand.


Hearing of November 13, 2014 (Live-blog - Official transcript)

During this hearing, the witness W.K. said that Eikonal was a one of a kind operation, there was targeted collection from traffic that transited Germany from one foreign country to another.

This was focussed on Afghanistan and anti-terrorism. Selected data were collected and forwarded to NSA. The internal codename for Eikonal was Granat, but that name wasn't shared with NSA. There was even a third codename.

For Germany, Eikonal was useful because it provided foreign intelligence for protecting German troops and countering terrorism. The NSA provided better technical equipment that BND didn't had. In return, BND provided NSA with data collected from transit traffic using search profiles about Afghanistan and anti-terrorism. BND was asked to cooperate because NSA isn't able to do everything themselves.

Eikonal provided only several hundred useful phone calls, e-mail and fax messages a year, which was a huge disappointment for NSA. This, combined with the fact that it proved to be impossible to 100% guarantee that no German data were collected and forwarded, led BND to terminate the program.

For Eikonal, the cable traffic was filtered by using selectors provided by both NSA and BND. Although not all selectors can be attributed to a particular country and there may have been up to several hundred thousand selectors, witness W.K. said that BND was still able to check whether every single one was appropriate: only selectors that could be checked were used.



Hearing of December 4, 2014 (Live-blog - Official transcript)

During this hearing, BND-employee S.L., who was the project manager of operation Eikonal at BND headquarters, testified. He told that BND had rented two highly secured rooms of ca. 4 x 6 meters in the basement of a Deutsche Telekom switching center in the Frankfurt suburb Nied.

These rooms were only accessible for BND personnel and contained the front-end of the interception system, existing of 19 inch racks, with telecommunications equipment like multiplexers, processors and servers. These devices were remotely controlled from the headquarters in Pullach.*

Based upon analysis of public information about telecommunication networks, BND choose specific cables that would most likely contain traffic that seemed useful for the goals of the operation. It became clear that for redundancy purposes, cables only used 50% of their capacity. For example, 2 cables of 10 Gbit/s carried only 5 Gbit/s of traffic, so in case of a disruption, one cable could take over the traffic of the other one.



The switching center of Deutsche Telekom in Frankfurt-Nied
where some cables were tapped under operation Eikonal
(Screenshot: ZDF Frontal21 - Click to enlarge)


After a specific coax or fiber-optic cable had been selected, technicians of Deutsche Telekom installed a splitter and a copy of the traffic was forwarded to one of the secure rooms, where it was fed into a (de-)multiplexer or a router so the signal could be processed. After they got rid of the peer-to-peer and websurfing traffic, the remaining communications data, like e-mail, were filtered by selectors from BND and NSA.

The selected data were sent back to BND headquarters in Pullach over a leased commercial line, of which the capacity was increased after the internet collection became fully operational. From Pullach to the JSA in Bad Aibling there was a 2 Mbit/s line.

Timeframe

Eikonal started with access to a telephone cable (Leitungsvermittelt). Project manager S.L. told that the first cable was connected (aufgeschaltet) in December 2004, but that it's signal was too weak. Therefore, in January 2005, an amplifier was installed.

In February, March and April additional cables were connected, so telephony collection started in the spring of 2005. By the end of 2006, Deutsche Telekom announced that its business model for dedicated transit cables would be terminated, so in January 2007 the telephone collection ended.*

BND also wanted access to internet traffic (Paketvermittelt), for which the first cable became available by the end of 2005, but because the backlink was missing, collection was technically not possible. This was solved in 2006, and in the spring of 2006 a second cable was added, and they tested the front-end system and subsequently the filter systems until mid-2007 (Probebetrieb).

During this stage, data were only forwarded to the joint NSA-BND unit JSA after a manual check. Fully automated forwarding only happened from late 2007 until operation Eikonal was terminated in June 2008 (Wirkbetrieb).*

Legal issues

The collection of telephone communications from transit cables was done under the general authority of the BND Act, with details specified in the "Transit Agreement" between BND and Deutsche Telekom, which for the latter was signed by Bernd Köbele.

For the collection of internet data it was impossible to fully separate foreign and domestic traffic, so it couldn't be ruled out that German communications were in there too. Therefore, BND requested an order from the G10-commission, which, like the FISA Court in the US, has to approve data collection when their own citizens could be involved.

A G10-order describes the communication channel (Germany to/from a specific foreign country) that BND is allowed access to, the threat profile and it also authorizes the search terms that may be used for filtering the traffic.*

Such an order allows the collection of G10-data (communications with one end German), which were processed within BND's separate G10 Collection program. As a bycatch, this G10-interception also yielded fully foreign traffic (Routine-Verkehre), which was used for operation Eikonal:




Some employees from Deutsche Telekom and from BND had doubts about the legality of this solution, which seemed to use a G10-order as a cover for getting access to fully foreign internet traffic.

Eventually, the federal Chancellery, apparently upon request of the BND, issued a letter saying that the operation was legal. This convinced the Telekom management and the operation went on. It didn't become clear under what authority this letter was issued.

After BND had learned how to collect internet traffic from fiber-optic cable, it applied for G10-orders to intercept (one end German) communications from 25 foreign and domestic internet service providers in 2008. This time these cables were being tapped at the DE-CIX internet exchange, which is also in Frankfurt.

Results

The collection under operation Eikonal resulted in only a few hundred intelligence reports (German: Meldungen) a year, each consisting of one intercepted e-mail, fax message or phone call. These were burned onto a CD to hand them over to NSA personnel at the JSA.*

According to S.L., metadata (containing up to 91 fields) were "cleaned" so only technical metadata (Sachdaten) were forwarded to the JSA, where they were used for statistical and analytical purposes.

Personal metadata (personenbezogene Daten), like e-mail and IP addresses were not shared. Technical metadata are for example used to identify the telecommunication providers, transmission links and the various protocols.


Hearing of December 18, 2014 (Live-blog - Official transcript)

During this hearing, a talkative general Reinhardt Breitfelder, head of the SIGINT division from 2003-2006, confirmed many of the details from the earlier hearings of his subordinates. He also gave impressions of the dilemmas in dealing with the NSA and what to do with the equipment they provide.


Hearing of January 15, 2015 (Live-blog - Official transcript)

In this hearing, the commission questioned two employees from Deutsche Telekom (Harald Helfrich and Wolfgang Alster), but they provided very little new information, except for that Deutsche Telekom personnel only knows between which cities a cable runs, but they don't know what kind of traffic it contains - they are not allowed to look inside.



A room where hearings of the parliamentary committee take place
(photo: DPA)

 

Disclosures from Austria

On May 15, 2015, Peter Pilz, member of the Austrian parliament for the Green party, disclosed an e-mail from an employee of the Deutsche Telekom unit for lawful intercept assistance (Regionalstelle für staatliche SonderAuflagen, ReSa), who notified someone from BND that apparently a particular fiber-optic cable had been connected to the interception equipment. The e-mail describes this cable as follows:

Transit STM1 (FFM 21 - Luxembourg 757/1), containing 4 links of 2 Mbit/s:

Channel 2: Luxembourg/VG - Wien/000 750/3
Channel 6: Luxembourg/CLUX - Moscow/CROS 750/1
Channel 14: Ankara/CTÜR - Luxembourg/CLUX 750/1
Channel 50: Luxembourg/VG - Prague/000 750/1

STM1 stands for Synchronous Transport Module level-1, which designates a transmission bit rate of 155,52 Mbit/second. A similar multiplexing method is Wavelength-Division Multiplexing (WDM) commonly used in submarine fiber-optic cables. The latter having a much larger capacity, generally STM-64 or 9,5 Gbit/second.

The number 757 is a so-called Leitungsschlüsselzahl (LSZ), which denotes a certain type of cable. In this case it stands for a channelized STM-1 base link (2 Mbit in 155 Mbit), which seem to be used for internal connections.

According to the meanwhile updated LSZ List, the number 750 stands for a "DSV2 Digitalsignal-Verbindung 2 Mbit/s", which is a digital signal path.

The cable mentioned in the e-mail therefore only has a small capacity, which seems to indicate that NSA and/or BND selected it carefully.

FFM 21 stands for "Frankfurt am Main 21", which according to Deutsche Telekom's network map is the name of the Point-of-Presence (PoP) located at its facility in the Frankfurt suburb Nied - the location where that Eikonal tapping took place.

This means we have a physical cable running between Luxembourg and the Deutsche Telekom PoP in Frankfurt, but containing channels to cities which are much further, so they have to connect to channels within other physical cables that run from Frankfurt to Moscow, Prague, Vienna and Ankara, respectively:



As the e-mail is from February 3, 2005, it must relate to telephone collection, because for Eikonal, the first cable containing internet traffic only became available by the end of that year.


The Transit agreement

On May 18, the Austrian tabloid paper Kronen Zeitung published the full "Transit Agreement" (pdf) between BND and Deutsche Telekom, in which the latter agreed to provide access to transit cables, and in return will be paid 6.500,- euro a month for the expenses. The agreement came into retrospective effect as of February 2004.

This disclosure got little attention, but is rather remarkable, as such agreements are closely guarded secrets. The Transit agreement existed in only two copies: one for BND and one for Deutsche Telekom.

It is not known how Pilz came into possession of these documents, but it seems the source must be somewhere inside the German parliamentary investigation commission. They are the only persons outside BND and Deutsche Telekom who, for the purpose of their inquiry, got access to the agreement and the other documents.

Leaking these documents to Pilz seems not a very smart move, as it will further minimize the chance that the commission will ever get access to the list of suspicious NSA selectors.


Country lists

On May 19, Pilz held a press conference (mp3) in Berlin, together with the chairman of the Green party in Luxembourg and a representative of the German Green party. Here, Pilz presented a statement (pdf), which includes the aforementioned e-mail, 10 questions to the German government, and two tables with cable links to or from Austria and Luxembourg:



Lists of links that apparently were on a priority list of NSA.
LSZ = Leitungsschlüsselzahl (cable type indentifier);
Endstelle = Endpoint; Österreich = Austria.
(Source: Peter Pilz - Click to enlarge)



According to Pilz, the full list contains 254 (or 256) cable links. 94 of them connect EU member states, 40 run between EU members and other European countries like Switzerland, Russia, Serbia, Bosnia-Herzegovina, Ukraine, Belarus and Turkey. 122 links connect European countries with nations all over the world, with Saudi Arabia, Japan, Dubai and China being mentioned most.

The country which most links (71) run to or from is the Netherlands. The list for that country was disclosed by Peter Pilz during a press conference in Brussels on May 28, 2015. The US, the UK and Canada are not on the list, although there were apparently 156 links from/to Britain too.

Update:
On June 25, 2015, the Dutch telecommunications provider KPN announced the results of its inquiry into the alleged tapping of its cables. It was very difficult to identify the channels in the list because meanwhile KPN's whole network had been restructured. Eventually it became clear the connections (being channels within cables and KPN only being responsible for the first half until Frankfurt) had been rented out under telephony wholesale contracts, so it was impossible to trace individual customers or users.
 
Additional details

On June 5, 2015, Peter Pilz held a press conference in Paris, where he presented a statement (.docx) containing a list of 51 transit links to or from France. Interestingly, this list now also includes some additional technical identifiers for these links, which were apparently left out in the earlier ones:



First part of the list with links related to France
(Source: Peter Pilz - Click to enlarge)


On June 29, 2015, Peter Pilz presented a similar detailed list (.pdf) of 28 transit links to and from Poland.

According to the updated LSZ List, the new codes in these lists stand for:

- 703: VC3 Virtual Container connection with 48,960 MBit/s
- 710: (not yet known)
- 712: VC12 Virtual Container connection with 2,240 MBit/s
- 720: (not yet known)
- 730: (not yet known)

VC3 and VC12 are from the Synchronous Digital Hierarchy (SDH) protocol to transfer multiple digital bit streams synchronously over optical fiber. This has the option for virtual containers for the actual payload data. VC3 is for mapping 34/45 Mbit/s (E3/DS3) signals; VC4 for 140 Mbit/s (E4); VC12 for 2 Mbit/s (E1).

The new identifiers in this list stand for: O-nr.: Ordnungsnummer; GRUSSZ: Grundstücksschlüsselzahl; FACHSZ: Fachschlüsselzahl.

No information about these identifiers was found yet, but by analysing the data in the list, it seems that the FACHSZ codes are related to a telecom provider. France Telecom for example appears with FACHSZ codes CFT, VPAS, VCP3, VB5 or 0.

The GRUSSZ number identifies a particular city, with the first two or three digits corresponding with the international telephone country codes. The last two digits seem to follow a different scheme, as we can see that a capital always ends with "10":
Paris = 33010
Lyon = 33190
Reims = 33680
 Brussels = 32010
Prague = 42010
Oslo = 47010
 Warsaw = 48010
Poznan = 48020
Moscow = 70010
It's possible that these are just internal codes used by Deutsche Telekom, as internationally, connections between telephone networks are identified by Point Codes (PC). From the Snowden-revelations we know that these codes are also used by NSA and GCHQ to designate the cable links they intercept.



NSA or BND wish lists?

Initially, Peter Pilz claimed these links were samples from a priority list of the NSA, but on May 27, he said in Switzerland, that the list was from BND, and was given to NSA, who marked in yellow the links they wanted to have fully monitored.

The German parliamentary hearings were also not very clear about these lists. On December 4, project manager S.L. confirmed that NSA had a wish list for circuit-switched transit links, but in the hearing from January 15 it was said that there was a "wish list of BND" containing some 270 links. And on March 5, former SIGINT director Urmann said he couldn't remember that NSA requested specific communication links.

Maybe the solution is provided by the Dutch website De Correspondent, which reports that there is a much larger list (probably prepared by BND) of some 1000 transit links, of which ca. 250 were marked in yellow (probably those prioritized by NSA).


Whose cables?

Media reports say that these cables belong to the providers from various European countries, but that seems questionable. As we saw in the aforementioned e-mail, it seems most likely that the lists show channels within fiber-optic cables, and that the physical cables all run between the Deutsche Telekom switching facility in Frankfurt and the cities we see in the lists.

In theory, these cables could be owned or operated by those providers mentioned in the lists, but then they would rather connect at a peering point like the DE-CIX internet exchange, instead of at the Deutsche Telekom switching center. Deutsche Telekom runs its own Tier 1 network, a worldwide backbone that connects the networks of lower-level internet providers.



Simplified structure of the Internet, showing how Tier 1, Tier 2 and Tier 3 providers
transit data traffic in a hierarchial way and how Tier 2 providers exchange
traffic directly through peering at an Internet eXchange Point (IXP)
(diagram: Wikimedia Commons - click to enlarge)


Questions

It is not clear how many of the over 250 links on the list were actually intercepted. We only know that for sure for the STM-1 cable with the four channels described in the aforementioned e-mail from Deutsche Telekom to BND.

Strange is the fact that during the parliamentary hearings, most BND witnesses spoke about "a cable in Frankfurt", which sounds like one single physical cable, whereas the disclosures by Peter Pilz clearly show that multiple channels must have been intercepted.

Update:
During the commission hearing of January 29, 2015, BND technical engineer A.S. said that under operation Eikonal, telephone traffic came in with a data rate of 622 Mbit/s. This equals a standard STM-4 cable, which contains 252 channels of 2 Mbit/s. This number comes close to the channels on the "wish list", but it seems not possible that those were all in just one physical cable.

Another question is whether it is possible to only filter the traffic from specific channels, or that one has to have access to the whole cable.

It should be noted that not the entire communications traffic on these links was collected and stored, but that it was filtered for specific selectors, like phone numbers and e-mail addresses. Only the traffic for which there was a match was picked out and processed for analysis.


Possible targets

Based upon these documents, Peter Pilz filed a complaint (pdf) against 3 employees of Deutsche Telekom and one employee of BND for spying on Austria, although at the same time he said he was convinced the NSA was most interested not in Austrian targets, but in the offices of the UN, OPEC and OSCE in Vienna.

Apparently he didn't consider the fact that Eikonal was part of the RAMPART-A umbrella program, which is aimed at targets in Russia, the Middle East and North Africa. Many cities mentioned in the disclosed lists seem to point to Russia as target, and project manager S.L. testified that Eikonal was mainly used for targets related to Afghanistan, which fits the fact that there are for example 13 links to Saudi Arabia.

Green party members from various countries claimed that this cable tapping was used for economical or industrial espionage, but so far, there is no specific indication, let alone evidence for that claim.



Links and sources
- LeMonde.fr: Deutsche Telekom a espionné la France pour le compte de la NSA
- Tagesschau.de: Europa verlangt Aufklärung von Berlin
- DeCorrespondent.nl: Er is geen enkel bewijs dat de Nederlandse kabels zijn afgetapt
- Volkskrant.nl: 71 KPN-internetverbindingen afgetapt door geheime diensten
- NRC.nl: Duitse BND tapte tientallen internetverbindingen KPN af
- DerStandard.at: BND-NSA-Affäre: Laut Pilz auch Spionage in Belgien und Niederlanden
- Golem.de: Telekom und BND Angezeigt: Es leakt sich was zusammen
- Zeit.de: Daten abfischen mit Lizenz aus dem Kanzleramt

Tuesday, May 12, 2015

German BND didn't care much about foreign NSA selectors

(UPDATED: June 23, 2015)

Over the last couple of weeks, the German foreign intelligence agency Bundesnachrichtendienst (BND) was accused of helping the NSA by carelessly or even deliberately entering selectors used for spying on foreign targets in the German satellite interception system at Bad Aibling.

Here, recent outcomes of the German parliamentary inquiry will be combined with information from the various press reportings, in order to provide a more integrated picture of what happened over the past years.

It becomes clear that BND did everything that seemed reasonable to prevent that German data were passed on to the Americans, but that they didn't really care about whether NSA collected communications from other European countries.

We also learned new things about the selectors that are used for filtering the communications traffic, but it's still not fully clear to what extent BND is able to prevent German internet data being collected.

Information from the parliamentary inquiry hearings is derived from the live blog provided by the German digital rights website Netzpolitik.org.





The BND satellite intercept station at Bad Aibling, Germany
(Photo: AFP/Getty Images - Click to enlarge)


 
The context

This latest affair started on April 23, when the German magazine Der Spiegel reported that NSA apparently spied upon European and German targets for years, with the knowledge of the German foreign intelligence agency BND.

Other news reports inflated this to BND deliberately helping NSA in spying on these targets illegally, which led opposition leaders accusing the German government of treason. This although by then there was no clear evidence, only sometimes confusing and not always very accurate press reports.


Committee hearings

Meanwhile there's somewhat more clarity, also because last Thursday, May 7, the parliamentary committee investigating NSA spying and cooperation with BND (German: NSA UntersuchungsAusschuss, NSAUA) questioned the BND employees R.U., D.B. and Dr. M.T. (initials not of their real names, but of cover names!) who were involved in this issue.

The day before, May 6, the regular parliamentary intelligence oversight committee (Parlamentarisches KontrollGremium, PKGr) heard in a classified meeting BND president Gerhard Schindler and Thomas de Maizière, currently the Interior Ministor, but previously responsible for intelligence affairs at the Chancellery.

Update:
On May 20, the NSAUA committee heard three additional BND employees, and on May 21, also Hartmut Pauland, the former head of BND's SIGINT directorate, and BND president Gerhard Schindler. They provided interesting and clarifying details which are added to this article and are marked "Update #2" and "Update #3" respectively.



BND president Gerhard Schindler just before he testified before
the parliamentary investigation committee on May 21, 2015
(Click to enlarge)



The cooperation between NSA and BND

The cooperation between NSA and BND which is at stake here, started with a Memorandum of Agreement (MoA) signed on April 28, 2002, in which both parties agree on joint espionage areas and targets, such as counter-terrorism, the battle against organized crime and against proliferation of weapons of mass destruction.
Update #3:This Memorandum, classified as Top Secret, is an extensive document, with 5 annexes, describing in detail the regions that should and should not be monitored. For reasons unknown, and also unknown to current BND president Schindler, these guidelines were never converted into internal regulations for the personnel that had to work on this.

Two years later, NSA abandoned its Bad Aibling Station for satellite interception, that under the codename GARLICK was part of the ECHELON network. Most of the facilities, including nine of the large satellite dishes hidden under white radomes, were handed over to BND.

In return, BND had to share the results from its satellite collection with the NSA. For this cooperation the Joint SIGINT Activity (JSA) was set up, consisting of personnel from both NSA and BND. The Americans provided most of the equipment. The JSA was located at the nearby Mangfall Barracks and was closed in 2012. According to BND vice-president Müller, his agency took over the software and hardware that NSA left behind in 2013, and checked it for backdoors.

Besides the satellite interception, Bad Aibling was also involved in cable tapping, but only for operation Eikonal (2004-2008), which was limited to cables from Deutsche Telekom in Frankfurt.



Google Maps view of the Mangfall Barracks in Bad Aibling, Germany.
The building in the upper left corner could be the BND facility,
and the one with the white roof the NSA's "Tin Can".

Selectors

For the satellite interception in Bad Aibling, initially some 4 out of 5 selectors came from the Americans, the rest were German (currently still 4:1). NSA started providing the Germans with telephony selectors in April 2005, followed in 2007 with selectors for IP communications. They are classified as Secret* and most of them were related to Afghanistan.

According to Süddeutsche Zeitung, NSA provided BND with roughly 690.000 phone numbers and 7,8 million internet identifiers between 2002 and 2013. That is an average of something like 60.000 phone numbers and 700.000 internet identifiers a year, or 164 phone numbers and over 1900 internet identifiers each day.

Just recently, BND found two additional selector databases in the legal division of its SIGINT directorate: one containing some 400.000 selectors from early 2005, including some related to European governments, but it couldn't be determined whether selectors were rejected.

A second database contains 59.000 selectors from September 2006 till early 2008. 400 were marked as disapproved. Both lists contain phone and fax numbers as well as e-mail addresses, but no IP addresses. They don't include German companies or phone numbers starting with the German country code 0049.

Such selectors generally include phone and IMEI numbers, e-mail, IP and MAC addresses of computers and tablets, but also other kinds of internet identifiers, like names, nicknames and chat handles. These are called "hard selectors". It is not known whether also "soft selectors" like keywords or maybe even cookies and malicious code signatures were also used in this cooperation.
 
Update #2:To the surprise of the commitee, witness W.O. testified that the category "IP selectors" does not include IP addresses, but denotes e-mail addresses and other internet communication identifiers, for example for messaging.

In general, for one target there are multiple selectors (German: Telekommunikationsmerkmale (TKM) but also Datenbegriffe or Suchbegriffe) like phone numbers or e-mail addresses. For the latter there can be multiple permutations, like using "%20" instead of a dot. The witness never saw the use of wildcards, like *@example.com.

Already in 2005, there were separate databases for phone and internet selectors, which were newly set up in 2001. All selectors were first put in the database, and after they were checked, the ones that were rejected, were marked as inactive. So with all the old selectors staying in, and more and more new selectors came in, the database expanded rapidly.

Until 2012, the NSA sent the selectors in the form of a so-called "equation", which appears to be a record containing name(s), phone number(s) and e-mail address(es). An equation can contain up to one hundred selectors used by or related to a particular target. Besides phone numbers and e-mail addresses, an equation also contains the different ways of spelling and technical permutations thereof.
Update #3:According to president Schindler, e-mails can have up to 20 permutations, each of which is a separate selector, which explains the large numbers. He gave the example of gerhardschindler, gerhard.schindler, etc. However this seems a simplification, as such variations can of course belong to different persons with the same name.

Because of this, when BND rejected say a phone number, BND employees in Bad Aibling had to ask NSA to remove that number from the equation, or else the other selectors in that equation were rejected too. It's always the full selector profile that has to be activated for collection. Until 2011 NSA saw all the selectors that were rejected by BND because of this.

As of 2011 these equations were split up and phone and internet selectors were each put in separate databases. This made it possible to reject individual selectors. Then the computer system combines these parts to their proper equations, which can now have for example a rejected phone number alongside an approved e-mail address. But if one part is disapproved, such an equation will not be forwarded to the collection system.

 
How BND checks NSA selectors

The selectors provided by NSA were picked up by BND employees at Bad Aibling from an NSA server a few times a day. Initially their number was not very large. They were for example on Excell sheets which were checked manually at Bad Aibling.
Update #2: This check was only for the so-called G10-compliance, which means that selectors related to German citizens and corporations were taken out. Somewhere in 2005, BND also began to check for German interests, the meaning of that was determined by unit T2AB, which conducts these selectors checks.

Apparently talking about the Eikonal operation, witness D.B. explained the committee that in the testing phase, one BND employee did this on his own, which led to a delay of one day. In 2007 NSA wasn't satisfied by that and wanted the results in real-time.

 
3-stage filtering: DAFIS

Later, the number of selectors increased to a level that couldn't be checked by hand anymore. A new procedure was set up, in which, since June or August 2008, Bad Aibling personnel sent over the selectors to unit T2AB at the BND headquarters in Pullach once a week, without further inspection (until 2011 there was also a rarely used manual Emergency Approval).

At the headquarters, the selectors are checked in an automated process of 3 stages called DAFIS (probably the abbreviation of DatenFilterSystem):

1. A negative filter which filters out e-mail addresses ending with .de and phone numbers starting with 0049, but most likely also ranges of IP addresses assigned to Germany.

2. A positive filter consisting of a list of foreign phone numbers and e-mail addresses used by German citizens, for example businessmen, journalists, but also jihadis. Therefore, this relatively large list contains a few thousand numbers that will also not be monitored.

3. A filter to sort out selectors that collide with German interests. Witnesses heard by the committee wouldn't publicly explain how this works, but maybe in this stage selectors for European military contractors in which Germany participates (like EADS and Eurocopter) are filtered out.
Update #3: Former SIGINT director Pauland confirmed that this stage includes names of companies (also from other European countries when there's a German participation), but also names of German politicians (although not the names of the chancellor, members of parliament and EU commissioners), and newly added top level domains and country codes are blocked here too. These names were not added systematically. The DAFIS filter system is used for all collection facilities. For metadata this filter is applied after they have been collected from specific links.

The only regular manual check is for false positives, because for example SIM cards can have an IMEI number that also starts with 49 (the telephone country code for Germany).

Although this filtering was considered 99,99% accurate, the witness R.U. admitted in the hearing on May 6 that this method is not always able to prevent German communications being intercepted, for example when a German citizen uses an Afghan phone number and/or is calling locally in Afghanistan. Such numbers would not be rejected for tasking, and there's also no system that filters out spoken German language.

BND also prevented that communications of American citizens would be collected, but the witnesses didn't explained how that was done.


How to determine nationality?

During an earlier hearing, BND lawyer Stefan Burbaum said that in rare cases a conversation first had to be collected and listened to in order to determine whether the contents are under constitutional protection or not.

Likewise it is impossible to determine the nationality of the person using an e-mail address like for example "redgoose1432@hotmail.com" without further circumstancial information. Even the content isn't always decisive.

We know that NSA analysts have to determine a "foreignness factor" for every selector, to exclude that it belongs to an American. For BND however it's impossible to automatically check whether such a mail address could belong to a German.

Witness R.U. reminded that such cases are rather speculative, because generally selectors like phone numbers are only tasked when they have a connection to a known suspect or target.

Update #3: Former SIGINT director Pauland said that selectors can be attributed to a particular country by for example a telephone country code, the extension of an e-mail address, a mobile phone cell-ID, or the IP address which is contained in metadata of certain messenger services.
He also explained that metadata include all data that are not content: not only an address, but also technical data that are generated automatically, and they can also include browser-specific features like the language.


How to check internet selectors?

During most of the hearings for the parliamentary inquiry, the witnesses mainly spoke about (selectors for) intercepting telephone calls, and they weren't questioned about how internet communications are filtered.

This seems to be a missed opportunity, because for the latter it is much more difficult to sort out domestic communications. Phone numbers always start with a country code, but on the internet people use many kinds of identifiers that are not easily attributable to a specific country.

It would have been interesting to know how BND thinks they can prevent for example MAC addresses of devices used by Germans being monitored, or to what extent it is possible to determine the nationality of people behind nicknames. This is important, not at least because there are far more selectors for IP traffic than for telephony.

Update #2: The way this is done became more clear during the hearings of May 20 and 21, when we learned that selectors come in "packets" that seem to include all known selectors for a particular target.
Witness W.K. for example explained that for each target, there are multiple selectors, so when at least one selector can be attributed to a specific country, that also applies to the other selectors.


Positive filtering

It seems that BND tries to solve this issue with the positive filter, using a list of foreign identifiers used by German citizens. However, keeping such a list up-to-date would almost require an intelligence operation itself, but maybe they take a shortcut by requesting the phone numbers and e-mail addresses of Germans abroad from for example the foreign ministry, chambers of commerce and press organisations.

This seems doable for Germans, but it's obvious that this is impossible for companies and citizens from other European countries. This explains why apparently some NSA selectors for European companies made it through BND's selection system.


Economical espionage?

This doesn't automatically means NSA was (trying to) conducting economical or industrial espionage. According to Süddeutsche Zeitung, there are only very few indications for that. The paper says NSA was mainly interested in certain companies because they were looking for illegal (arms) exports.

For example, the e-mail address of an Airbus employee who was probably targeted by NSA, reportedly belongs to someone who is responsible for applying for arms export licences, which shows that targeting commercial companies can very well have valid foreign intelligence reasons.

On May 13, the head of Germany's domestic security service BfV, Hans-Georg Maassen, said that he has no evidence that the United States carried out industrial espionage in his country. The same was said by BND president Schindler, when he testified before the parliamentry commission on May 21, 2015.



An operations center room in the former BND headquarters in Pullach
(Photo: Martin Schlüter - Click to enlarge)


 
Discovery of suspicious selectors

Already in 2005, a BND employee discovered that among the selectors provided by NSA (at that time also used for the cable tapping under operation Eikonal), there were indentifiers for the European defense contractors EADS and Eurocopter (both now part of Airbus Group).

These companies have no protection under the German constitution, but it was considered that such information shouldn't be forwarded automatically. Selectors for French government officials were discovered somewhat later, according to witness D.B. on May 7.

Then in 2008, a BND official informed the Chancellery saying that NSA was apparently going after its own interests in Europe too. At least by then, BND started sorting out suspicious NSA selectors and put them in a separate database. Only in 2010 and 2011 three suspect things from 2005, 2006 and 2007/2008 were reported to the BND top management.

Update: During the hearing of June 12, 2015, former BND president Ernst Urhlau said that after it was discovered that NSA was apparently interested in some European targets too, the Americans were asked what that was about, and they apologized for that and said that it wouldn't happen again.


Storing rejected selectors

The check on the selectors took place at BND headquarters, but after that, they were sent back to Bad Aibling, where they were either entered into the collection system or stored in the rejected selectors repository (German: Ablehnungsdatei, sometimes also Ausschussliste).
Update #3:Actually there are two separate tasking systems: the main system is for BND's own selectors, and another, unique one, is for the selectors from NSA. The latter is only used in Bad Aibling, the main system is used at all BND collection facilities, so in Bad Aibling there were two separate tasking systems.

Although it could be interesting to know what NSA looks for but didn't pass BND filters, witness D.B. said this database isn't routinely looked at. He also said that NSA is informed about the selectors that have been rejected, which was apparently no problem for them.

Storing the rejected selectors was said to be useful because when NSA sends a suspicious selector again, it can be sorted out by checking against this list. Approved selectors are also sometimes marked as inactive, for example when a foreign extremist travels into Germany. Then BND monitoring has to stop, but when he leaves the country, the selector is activated again.



Overview of the dataflow for the NSA-BND cooperation at Bad Aibling
(Click to enlarge)


40.000 rejected selectors

Until 2013, the Ablehnungsdatei was filled with up to 38.000 NSA selectors which therefore didn't make it into the collection systems. With the 2000 selectors sorted out by Dr. T. (see below) this makes number of 40.000 selectors the press reported about, which is about 0,47% of the total number of selectors provided by the Americans.

Initially, Der Spiegel reported that these 40.000 were found through an investigation in the Fall of 2013, suggesting they had been active all the time and that thereby, BND enabled NSA to illegally spy on some 40.000 targets.

Given the criteria of BND's 3-stage filter system, these 40.000 must include NSA selectors that either have a German country code, a foreign identifier used by a German citizen or entity, or a match with the mysterious "German interests" criteria.

We don't know how many selectors were rejected for each of these stages, but we can assume that in a number of cases NSA did sent identifiers for targets that were recognizable as German. For selectors rejected in the second stage, NSA may not have known that a particular identifier was used by a German, something that BND could probably find out easier.

We also don't know how these 40.000 are divided among phone and internet selectors, which can also make a big difference, as it is much easier to attribute phone selectors to a particular country than it is for internet identifiers. Opposition leaders are demanding that the parliamentary investigation committee can see the list, but the government said they are still negotiating with NSA about this.



Office room in the former BND headquarters in Pullach, used by
an employee who cleary is a hardcore fan of Elvis Presley
(Photo: Martin Schlüter - Click to enlarge)


Investigating active selectors

Early August 2013, just a few months after the start of the Snowden revelations, BND Unterabteilungsleiter D.B. asked technical employee Dr. M.T. to take a look at the active NSA internet selectors to see what types of identifiers they contain and whether it could be determined what regions (Interessensschwerpunkte) NSA was interested in. This was the first systematic check since 2005(!).

For that, Dr. T. was provided with a copy of the database containing all selectors used in Bad Abling. This database copy was stored on a separate computer, because ordinary work stations couldn't process such a large dataset.

To his surprise, he found selectors that seemed politically sensitive. This investigation took about four weeks and resulted in some 2000 suspicious selectors. Dr. T. put them in a separate database, of which a single copy was printed out. These selectors were still active at that time, unlike the 38.000 which were prevented from being activated.

His copy of the database containing all selectors was deleted after the job was done. The one with the 2000 sorted out by Dr. T. wasn't found back after he had returned the dedicated computer, just like the list that had been printed out.
Update #3:Apparently, 40% of the selectors investigated by Dr. T. could not be attributed to a specific country.



Overview of the BND employees involved in the affair of the NSA selectors
(Click to enlarge)


Suspicous selectors deactivated

Immediatly after finding suspicious selectors, Dr. T. informed his superior Referatsleiter H.K., who reported this to Unterabteilungsleiter D.B. Around mid-August 2013, D.B. called the unit in Bad Aibling and ordered Dienststellenleiter R.U. to deactivate (although press reports call it "delete") the suspicious selectors in the phone and internet tasking databases (Steuerungsdatenbanken) and put them in the Ablehnungsdatei.

Meanwhile, D.B. had received the printed list with the 2000 selectors, consisting of a large number of sheets of paper, from Dr. T., and he sent this list to R.U. by courier. Using some specific criteria, it was then possible to remove the suspicious selectors. Strangely enough, D.B. thought all this not to be relevant enough to report to the BND president or to the Chancellery.

Der Spiegel reported that in the hearing behind closed doors on May 6, BND president Schindler said that the list of 2000 selectors almost exclusively contains e-mail addresses, not of companies, but mainly of European politicians, EU institutions and government agencies.

The reason for this result is clear now: e-mail addresses because Dr. T. only investigated internet selectors, and of European governments because BND didn't filter those out - according to BND president Schindler because they expected that NSA would comply with the Memorandum of Agreement, that prohibits selectors for European targets.

At least the fact that the list contains no German addresses seems to confirm that preventing German selectors from being monitored was successful, and that therefore there's no evidence that BND helped NSA in spying on German citizens, corporations or government officials.


Another investigation?

According to a report by Der Spiegel, BND employee R.U. was instructed on August 14, 2013 to "delete" some 12.000 search terms. These were apparently the outcome of an investigation in which BND's database with NSA selectors had been searched using terms like "gov", "diplo" and "bundesamt" (initially in some press reports erroneously presented as search terms provided by NSA).

This search had resulted in 12.000 hits (which doesn't necessarily means an equal number of selectors). The tabloid paper Bild am Sonntag reported that e-mail addresses containing the term "bundesamt" were targeted against Austrian government agencies and appeared in over 10 NSA selectors.

However, during the parliamentary inquiry, witness Dr. T. said that the three search terms mentioned by Der Spiegel and the number of 12.000 had nothing to do with his investigation. It's therefore unclear whether there was a second investigation, or that the press has mixed things up.

Update #2:During the committee hearing of May 20, it was confirmed that there was indeed a second investigation: mid-August 2013, R.U., head of the BND unit at Bad Aibling, ordered W.O. to check the NSA selectors for whether they were related to European governments. He only looked at e-mail addresses, because for other selector types it is too difficult to do such a check. W.O. also did research on the internet for his investigation, maybe for finding out the elements used in foreign government e-mail addresses.

Already after one day he found some, which were then deactivated ("deleted"). After that the search was continued for three weeks, adding additional search criteria. In the end this resulted in a few ten thousand selectors that were marked as rejected and then being deactivated.

W.O. only reported this to his immediate superior R.U., but at the Pullach headquarters, D.B. only heard of this second investigation and the subsequent deactivations in March 2015. SIGINT director Pauland wasn't even aware of both investigations before March 13, 2015. Then, a working group, led by BND lawyer Ms. F. was formed to investigate all these issues.

Additional updates:
On May 15, 2015, Der Spiegel reported that it seems some 25.000 of the 40.000 rejected selectors had actually been active.
Maybe these 25.000 are the ones that were deactivated after the second investigation conducted by W.O. If that's the case, then the 40.000 would consist of 2.000 found by Dr. T., 25.000 found by W.O., leaving 13.000 selectors that were sorted out before and/or after both investigations.

According to Klaus Dieter Fritsche, former official in the federal Chancellary responsible for BND, there are two printed versions of the list of the 40.000 selectors: one at BND and one in a safe in the Chancellery. However, this seems not to include the 2000 selectors sorted out by Dr. T.


BND takes measures

In November 2013, BND president Schindler issued a new internal regulation, saying that BND's own selectors may not include NATO and European targets anymore (no reason was seen to apply this to NSA selectors too). Reportedly e-mail addresses ending with .eu will now be blocked and the same has to happen for all European partners. We can assume this also applies to their telephone country codes.
Update #2:This regulation was apparently issued after chancellor Merkel came with her famous statement that it is not done to spy upon friends ("Ausspähen unter Freunden geht gar nicht") on October 24, 2013, following revelations that her mobile phone was targeted by NSA.
For blocking selectors related to European governments, there's a profile containing the e-mail extensions for all foreign government agencies.

However, this won't help European citizens, companies and organisations who are for example using phone numbers from outside Europe or mail addresses with a generic top level domain like .com, .org or .net. The new regulation is therefore most effective for preventing that communications of European government agencies will get caught in the filter systems.

Recently, BND asked NSA to provide a justification for every of their selectors. For telephone numbers, this was already practice,* but the Americans said that for internet selectors they needed more time. This led BND to stop the collection of internet data for the time being as of early May. Phone and fax data are still collected and forwarded.



BND president Schindler standing inside one of the huge golfball-like
radomes at the satellite intercept station Bad Aibling
(Photo: Reuters - Click to enlarge)
 

Results of the collection

After the approved selectors have been entered into the collection systems, all data for which there's a match with one or more selectors will automatically be picked out. These results are then converted into a readable format.

Matches for BND's own selectors are stored in a database: metadata went into VERAS and content into INBE. From there, analysts can see whether it is relevant for the foreign intelligence as required by the government. If not, the data are destroyed.

Many metadata collected in Bad Aibling were automatically forwarded to NSA, after passing the DAFIS filter system to sort out those related to Germans. According to the newspaper Die Zeit, BND collects about 220 million metadata each day, which is 6,6 billion a month. Up to 1,3 billion of these metadata are shared with NSA, an example being the 552 million metadata seen in a chart from the NSA tool BOUNDLESSINFORMANT.

Update #2:After the chart with the 552 million metadata was first published on July 29, 2013, the BND unit at Bad Aibling was in shock. They worked day and night and over the weekend to find out what had happened, and provide explanations of the technical circumstances in weekly reports, like for the regular parliamentary oversight committee.
After a week, BND was then able to issue a statement that these 552 million metadata were not collected by NSA, but by them, from crisis regions abroad.


Screenshot from BOUNDLESSINFORMANT, showing some 552 million telephone and internet
metadata that were shared with NSA between December 10, 2012 and January 8, 2013
(Click to enlarge)


Shortages

Content collected through selectors provided by NSA was also automatically forwarded after a final check by the DAFIS filter system, but here, BND personnel in Bad Aibling also took random samples to check whether it contained German data.

Because of shortages in personnel and technical capacity, BND employees were fully occupied with the results from their own selectors, and therefore had no time to take a closer look at what came out for NSA. They simply relied upon the initial selector check. Only when BND's own selectors didn't provide useful results, they would take a look at the results of the NSA selectors.


Selected communication links

One important fact that was largely overlooked in the reporting on this issue, but was pointed to by BND president Schindler and one of the witnesses, is that the Bad Aibling station only intercepts satellite links from crisis regions in the Middle East and Africa.
Update #3:During the hearing on May 21, Schindler specified this and said Bad Aibling collects data from all the countries where German forces are deployed and one other country he would not name. SIGINT director Pauland said BND is currently watching various crises around the world: Ukraine, IS, Boko Haram, Bundeswehr deployments, kidnappings, and Ebola; they are not spying on their own citizens.

Interception results therefore include for example phone calls between Afghanistan and Pakistan or communications from European companies and agencies with activities in the Middle East. This would also minimize the chance that German communications were being collected. BND selects which satellites and which communication channels from those satellite links are intercepted; NSA is said to have no influence on that.



No records kept

According to Der Spiegel, BND president Schindler said that his agency has no technical means to reconstruct which data were passed on to NSA as no records or statistics were kept on this. Earlier, BND employees also testified that their agency doesn't count the raw data that come in, only the end reports.

This means, that the lists of selectors can only show what NSA was interested in, but that we will probably never know what exactly the results from that collection were.

 
Update #1:

In an article of the newspaper Die Zeit from May 19, 2015, the Left party member of parliament Martina Renner says that in August 2013, there were between 8 and 9 million active selectors. Other sources say 8,2 million.

Earlier, Süddeutsche Zeitung reported that currently there are some 4,6 million active selectors, most of them for filtering internet communications and related to 1,267 million people and corporations.

If these numbers are correct, they would show a huge decrease of active selectors between 2013 and 2015. The hearings haven't provided clarity about this yet.

In Die Zeit, Martina Renner also said that BND didn't check all these selectors for whether they contained suspicious ones, but only looked at e-mail addresses for whether these contained parts like .de or names of German companies.

The reason for that seems to be that, according to Renner, there are more than 20 different kinds of selectors, and for 40% of the selectors (which would be over 3 million) it wasn't possible to attribute them to a particular country.
 
Update #2:

On May 20, 2015, the parliamentary investigation committee heard BND employees W.O., W.K. and D.B. about the issue of the selectors and the internal BND inquiries. New details from this hearing have been added to the relevant sections of this article. They are marked "Update #2".

In general, the witnesses from BND gave the impression that they don't look much further than the requirements and the responsibilities of their job. They just follow orders and that's it.
 
Update #3:

On May 21, 2015, the parliamentary investigation committee heard Hartmut Pauland, the former head of BND's SIGINT directorate, and BND president Gerhard Schindler. Details from this hearing have also been added and are marked "Update #3".

President Schindler admitted that the automated filtering of selectors was a mistake and that there were serious deficiencies in how this was handled internally. But he was also convinced that spying on European countries ("friends") isn't illegal, explicitly contradicting the opinion of three constitutional experts from the very first committee hearing.

Former SIGINT director Pauland said that with every newly disclosed Snowden-document, his people considered whether they were also capable of doing those things. In many things, NSA appeared to be way ahead of BND. Nowadays, signals intelligence is metadata-centric: from the metadata it's decided which communications are worth and useful to pick out for analysing their content.



Links and sources
- Offical page of the committee: 1. Untersuchungsausschuss ("NSA")
- SPD proposal for new legislation: Rechtsstaat wahren – Sicherheit gewährleisten! (pdf)
- Zeit.de: BND-Chef Schindler will nichts gewusst haben
- Netzpolitik.org: Interne Kommunikation: Wie der BND die „Weitergabe von Rohdaten in großem Umfang“ an die NSA verheimlicht (May 2015)
- Welt.de: Gezielter Angriff (May 2015)
- Zeit.de: BND liefert NSA 1,3 Milliarden Metadaten – jeden Monat (May 2015)
- Golem.de: Der Mann, der die brisanten NSA-Selektoren fand (May 2015)
- Netzpolitik.org: Untersuchungsausschuss: „Ich habe Weisung von oben empfangen und vollzogen“ (May 2015)
- Spiegel.de: Spionageaffäre: BND kann Daten-Weitergabe an NSA nicht rekonstruieren (May 2015)
- Sueddeutsche.de: BND half NSA beim Ausspähen von Frankreich und EU-Kommission (April 2015)
- FAZ.net: BND-Spionage-Vorwürfe: Spionieren und spionieren lassen (April 2015)
- Spiegel.de: Spying Together: Germany's Deep Cooperation with the NSA (June 2014)
- NSA-document about NSA counter-terrorism cooperation with BND and BfV (pdf) (2013)