The goal of the "Free and Open Source Security Audit" (FOSSA) pilot project is to increase security of Free Software used by the European institutions. The FSFE has been following the project since the early beginning in 2014. I am concerned that if the project stays on its current course the European Institutions will spent a large part of the 1 Million Euro budget without positive impact on the security of Free Software; and the result will be a set of consultancy reports nobody will ever read. But if we work together and communicate our concerns to the responsible people in the Parliament and the Commission, there might still be a valuable outcome.
The [FOSSA] project should result in a systematic approach for the EU institutions to ensure that widely-used key open source components can be trusted. The project will also enable the EU institutions to contribute to the integrity and security of key open source software. The EU-FOSSA project is managed by the European Commission's Directorate-General for Informatics (DIGIT). It was initiated by the European Parliament. (see Joinup.)
Since the European Parliament approved funding for FOSSA in 2014, the FSFE has been involved in the project. We gave input at the beginning, we suggested interview partners to the Everis – the consultancy company, which the European commission contracted for the project – and we were interviewed by them ourselves.
Unfortunately our experience from the participation leaves us concerned.
Our concerns
The whole analysis of Free Software which is published on Joinup is based on a small sample. They included 14+2 communities. From those they did interviews with only 5 (Apache – tomcat, Drupal, Free Software Foundation [!sic it was FSFE], Libre Office, OWASP Community). They did not contact several people from security teams of Free Software initiatives, whom we recommended and introduced to them.
The interviews which Everis conducted for the European Commission were quite strange: The questions were either very low level, or very academic questions about security. There was also no follow-up after the interviews. They had no good knowledge about Free Software when they started, and although we provided them with lots of information, when looking at the results this did not change.
There was little communication about the project until the recent publication of over 550 pages and a consultation about which program should be audited. In our comments from 8 July 2015 we recommended them to "Release Early, Release Often: Do not produce a huge report and publish this at the end of the project. Publish ideas, criteria, results, and next steps as soon as possible. Thereby you enable others to give you feedback." Publishing 550 pages after almost a year of silence does not allow to comment on misunderstandings and systematic errors they made early in the project – see below for some examples.
In general the projects is missing the step how security problems will be fixed. At the moment it proceeds in three phases: 1. The EU does code reviews 2. They provide the outcome to the Free Software communities by considering "industry standards for submitting code reviews" 3. Then the EU institution's user has more secure software. But this approach does not answer the question how those problems actually get fixed and by whom. For example where does the money and resources for the "communities" come from?
I cannot assess how good the analysis for the European institutions was done by consultancy companies Everis (Spain), KPMG (Italy) and Trasys (Belgium). But the deliverables for the Free Software communities include a lot of wrong factual information and shows a lack of understanding for Free Software.
Factual errors
Those are just some points I found by reading through the deliverables once. My guess is that it will take me at least a week to refute most of the problems in this report line by line. But maybe you can help to point out the most relevant ones, so we can summarise them for the European Commission.
Unfortunately the best one page summary about the Free Software competence of the 568 pages of material the consultants is page 46 with the "License Overview": some things are correct, but many are wrong or do not make sense.
But let us continue with some other examples. In the "Analysis of Software Development Methodologies Used in the FOSS Communities" there are also many errors or missing information which should have been easy to research. For example:
- Debian often gets "N/A" although we proposed them an interview partner from Debian's security team, and lots of the information could have been researched with public information.
- "Generate LTS (Long Time Support) Releases" has a "N/A" for Debian and Red Hat.
- OpenSSL is just used by two (Debian and Red Hat have a "N/A" again)
- According the the deliverable Red Hat is not using "OpenSSH".
- GnuPG is marked as not being used by several of them, and "JAVA/JAVA EE" is mentioned as related technology of GnuPG.
- Java, and PHP are "N/A" for Debian but python is checked. While Red Hat again has a "N/A" for all of them.
- Why does LibreOffice have so many question marks?:
Lack of understanding
-
In the deliverables they often write that Free Software communities consist of volunteers. It is sometimes mentioned that they are paid by companies, but it is not really considered further in the analysis.
-
It is not even clear if they understood how commercial Free Software works. At least this comment did not show understanding that there is no difference between commercial and non-commercial Free Software, when it comes to licenses: "Free for Open Source: Some communities provide their software as free for open source projects, meaning that projects which are commercial or proprietary in nature have to pay the license of the product."
-
The consultancy company focused on web services. They analysed Debian and Red Hat as if they are specific web tools. E.g. see recommendations like "Use PHP Snippets Sparingly and with Caution" or "SEO Best Practices" for GNU/Linux distributions.
Next steps
I am aware that those comments sound quite pessimistic. But I hope that if we give good feedback to the European Commission and the Parliament, and if they implement it, we might still be able to get some positive results for the security of Free Software. Crucial for that goal is that:
- The Commission communicates the status in digestible format and enable people to give feedback
- They publish the deliverables on a platform so people from the Free Software initiatives and companies can make annotations. Or at least publish the deliverables in an editable Open Standard format like ODT, to make it easier to give feedback.
- The Commission and the consultancy companies include more Free Software experts in the process. Especially the people from Linux Foundation's Core Infrastructure Initiative and the ones from Mozilla's Security Open Source Fund.
- The consultancy companies have to send short summaries to the projects interviewed so they can check if their summaries about their projects are accurate.
What you can do
As you are the real experts, now we need your feedback. What do you think about the deliverables themselves and what about our comments so far? I am especially interested if you benefit from the recommendations.
Send the feedback to the FSFE's discussion list, to me directly, or blog about it and sent me the link.
Afterwards I will sent an updated feedback to the Commission, and the members of the European Parliament Julia Reda and Max Andersson who initiated the pilot project.
Update 2016-07-11: Mirko Böhm, who was also involved in the project from the beginning, wrote down his comments on the deliverables and encourages you to give feedback, too.