Portugal not yet cleared of the ESF evaluation fallout, despite FCT president’s resignation
Until a few weeks ago, Miguel Seabra was president of the Portuguese Foundation for Science and Technology (FCT). He resigned on 7th April 2015, invoking personal reasons, as rumours about him having kept his position at Imperial College while being at FCT were beginning to circulate. Maria Arménia Carrondo, a researcher from the same laboratory as Seabra has just been appointed by the government as his successor. Portuguese scientists are waiting to see whether she will maintain the status quo or attempt to find constructive solutions. Above all, the Portuguese science community is now calling for increase in transparency and integrity.
In any event, Seabra’s position at the helm of the Portuguese funding agency was becoming untenable. He had been under tremendous pressure following the discredit shed on the FCT by a controversial evaluation of Portugal’s research units, outsourced to the European Science Foundation (ESF). The move was designed to select units that would need to be closed as part of a national austerity drive and due a reduction of European funds that may be used for research. A big question mark has been hanging over the quality of the evaluation since the Portuguese science community started to realise that, in many ways, those bound to survive the cut were not to do so on scientific merit.
Download a free PDF version of the article and spread it!Before Seabra resigned, an unexpected 40% cut on the funding of top ranked research units was announced. This follows the planned elimination of 50% of all research units in the country, made public in June 2014. In addition, FCT also announced further restrictions on doctoral and post-doctoral grants. This move will push many researchers into either unemployment or towards emigration, due to lack of alternatives.
Flawed evaluation
As the questionable conditions of the FCT evaluation emerged, the disastrous situation for the Portuguese research community became increasingly obvious to international observers. As it turns out, Spanish astrophysicist Amaya Moro Martin was right. In October 2014, she wrote an opinion piece in Nature, regarding science policy in Europe. She included a reference to Portugal saying: “Already reeling from budget cuts of 50% for universities and research centres, Portugal may now have to close half of its research units because of a flawed evaluation process supported by the ESF.”
Shortly after that, the head of science support of ESF, Jean-Claude Worms, wrote an email to Amaya, threatening her with a lawsuit. ESF backed off on this ridiculous intention a week later, perhaps realising that victimising a young researcher was not a good move. The ESF wikipedia entry sums up the impact the move has had on ESF’s reputation in the following terms: “ESF was very much discredited after a controversial evaluation of the Portuguese research centres.”
To say that the evaluation process is flawed is an understatement. It was a scam setup to justify the closure of half of the research units in the country and concentrate the major financial resources on a small number of units; incidentally, the units related to the current science policy makers in Portugal were awarded generous funding.
Some of the flaws have been identified by members of the Portuguese science community—with details described here and here. They include changing the rules when the process was well underway. And they also involve the use of incorrect bibliometric data. The Council of Rectors of Portuguese Universities (CRUP), in November 2014, protested with strong words, qualifying the evaluation as ‘a total failure.’ Three of the more outrageous issues identified, include the use of hidden quotas, an inadequate evaluation panel and the arbitrary allocation of funding.
Hidden quotas
But let’s get further into the details of what went wrong. The evaluation was carried out in two stages. Only the units proceeding to stage two were eligible for the large part of the funding available. The others will have no funding at all or funding at such a low level that it will not allow them to pursue any activity. In short, units that are not proceeding to stage two are condemned to disappear.
What nobody knew at first is that half of the research units were condemned even before the evaluation had started. After legal pressure FCT made public the contract with ESF. It then became obvious that ESF had been explicitly instructed to a priori exclude 50% of the units from stage two. The work plan for the panels spells it out in the following terms: “Stage 1 evaluation will result in a short list of half of the research units that will be selected to proceed to stage 2.” ESF complied. To this day FCT stills denies the existence of quotas, making their own very peculiar interpretation of what is written in the contract (please check for yourself). Stage one did not include site visits, as is mandatory by law. Sitting at their desks, ESF evaluators simply killed half of the Portuguese research units at a stroke of the keyboard
Light evaluation panel
However, this was not, by far, the only issue. When it comes to the adequacy of the review panels, the operation details of the evaluation speak for themselves. ESF set up a very small number of panels. For example, in the 2007 international evaluation 15 mathematicians, six physicists and seven chemists were gathered in three panels to analyse the units of the corresponding disciplines.
In 2014, the ESF formed a single panel for the three disciplines, with only 11 members (one engineer, three physicists, four chemists and three mathematicians). This panel, which decided to shut down several condensed matter physics labs in the country, had a single condensed matter physicist. In stage two, a team of evaluators visiting an architecture unit did not include a single architect and there were several other examples of clear mismatches between panels and the units they were supposed to evaluate.
Arbitrary funding
Furthermore, the most recent FCT funding allocation does not display a clear correlation with the ESF assessment, considering comparable unit sizes and laboratory intensity. There are 11 units classified as ‘exceptional,’ which is the highest possible grade. Of the five units with the highest funding awarded, only one was classified as exceptional. The unit with the highest funding per capita is in the humanities and is headed by a former chair of an ESF scientific committee.
So FCT asked ESF to make an assessment and, in the end, “modulated”—this is the actual FCT word —the output to award funds in an almost arbitrary way. This means that FCT called ESF as external institution to enforce predetermined funding intentions, regardless of scientific merit. ESF, who was paid a consultancy fee, was only too happy to oblige.
Out of the 322 research units being evaluated, 178 made it to stage two. Meanwhile, the majority of those, which did not make the grade, have appealed the decision, first to FCT, and then by demanding a new evaluation process. The latter is to be carried out by a different panel according to FCT’s own regulations. Four lawsuits are already underway. Out of the 178 “lucky ones,” 123 filed complaints regarding the second stage results, failing to see the logic of the process. As a final despicable move, FCT gave a three-day deadline for units to accept whatever funding befell them.
Portuguese scientists clearly have reasons to be concerned, but maybe they should not be the only ones. Miguel Seabra still remains president of Brussels-based research advocacy group Science Europe, whose aim is to influence European research policies. With current European commission winds blowing against fundamental research, having someone with such a track record in science management cannot be a good omen.
Professor of Physics, University of Coimbra , Portugal.
Can the FCT evaluation be rescued by the new management in place?
Your thoughts and opinions are valuable, feel free to use our simple comment section below.
Featured image credit: FCT
EuroScientist is looking for contributors!
If you would like to write guest posts in EuroScientist magazine, send us your suggestions of articles at office@euroscientist.com.
EuroScientist journal 
@G R Semin
Whatever…
Here is a summary of
the various “vacuous” points which reflect my opinion and which
were made in the previous posts, plus some other notes:
1. the latest
evaluation of Portuguese research units by the tandem FCT-ESF was a
scam designed to secure funding for some units at the cost of
eliminating others;
2. the rules of the
game were poorly designed and had, to align with the previous point,
to be changed several times when the evaluation was already in full
swing; [btw, this alone should be enough for Portuguese courts to
declare the whole process void]
3. for many units,
whether they made it to the second stage or not depended not on the
opinion of experts but on the decision of a member of the panel who,
in many cases, was not well equipped to make that decision and was
under very strong pressure to follow the quotas enforced by FCT; the
same applies to the decision to attribute the different marks in the
second stage;
4. the distribution
of funding did not follow the rules initially announced by FCT and
the percentage of the funding corresponding to each of the final
marks was set after the fact in order to keep within the budget; this
was completely predictable, since the method of giving a percentage
of what units ask for is not only idiotic but also impossible to
follow in practice;
5. apart from other
evidence, the effects of the above may be seen from a careful
analysis of the results; for instance, even before the contract
between ESF and FCT describing the 50% quota was made public, this
fact had already been discovered from an analysis of the data
available at that time;
6. this analysis
extends to other general issues such as the fact that the only
quantitative piece of data which showed any correlation to whether a
unit made it to the second stage or not was its size; one possible
interpretation of this is that this part of the evaluation was
extremely superficial;
7. the many
evaluation flaws of the system also transpire in other ways, as is
the case of some specific units; whether you like it or not, your
centre’s evaluation reflects the appalling funding strategy enforced
by FCT;
8. you may continue
arguing about whatever fallacies you want until you’re out of breath,
but the bottom line is the following: for many, if not most, units,
this evaluation was a lottery;taking that into consideration, under the conditions
of this evaluation, if people had had a choice, would they have
preferred to be evaluated by a panel where they knew one person (with
whom they were on good terms) and where at least one of the other
members was not fit to evaluate their unit, or by a panel where they
did not know anyone and none of the members was from a relevant area
[as happened to many other units]?
9. during the period
when units may protest, all the relevant information concerning the
evaluation, including the reports, may, by Portuguese law, be made
available to the other units upon request.
@ESF evaluation observer
Gün R Semin
I am not sure with whom I am having this virtual conversation
since the author of the blog does not sign with a name. Whoever the participant
to this exchange may be – this is turning out to be a ‘vacuous
exchange’. Let me continue with by adding some examples by my anonymous
partner to this exchange. First, let me provide an example provided by a
colleague of mine.
A burning cigarette was carelessly discarded.
Several acres of virgin forest were destroyed.
You could let these two statements stand as two
unrelated utterances. “But that’s not what you did, right?”
What is the difference between this example and
the examples this blogger uses to insinuate bias?
Here another example of a vacuous point about the use of English
phrases, such as ‘admittedly’, and I quote: “According to the Collins English Dictionary,
“admittedly” means “wittingly conceded”
which does sound a bit odd here, since you are not the author of the original
text, and, as far as I can tell, the author himself has not yet stated anything
in that direction. What actually happened was that you misinterpreted the text
and then proceeded to classify your own misinterpretation as being “completely
wrong”, using this to try and discredit other parts of the text – could this
also be classified as an association fallacy or does it have another name?”
This starts with an unfortunate and self-centered definition of ‘admittedly’ illustrative
of the type of rhetorical games straddling this vacuous dialogue [more about why
this dialogue is vacuous later].Let us start by rewinding. Admittedly is a term “… used to introduce a concession or
recognition that something is true or is the case: admittedly, the salary
was not wonderful, but the duties were light | this is admittedly an
extreme case.” – which translates to – and I quote my text again “admittedly,
the original text is not an example of clarity” meaning an exercise in
confusion, or meaning the ‘text is not an example of clarity to a reader’ (not
sure where the author of the blog comes into this except that the writing is not clear).
Do we want to continue quibbling? It’s a waste of time. The only contentious issue is
unjustified insinuations – but obviously, admissions are not on the agenda!
Admittedly, the author of the blog has insider information about
the inner workings of the FCT, which I do not have. It interesting to think
about how this privileged information was accessed. One-way is via the FCT
(very unlikely) and the second is sad even to consider. Whatever, if the blogger
were really well informed when attacking the center, then s/he would have to be
better informed about specific details during the evaluation process.
Admittedly, the FCT evaluation process does not win first
prize in a beauty contest, although the original intentions of the FCT may have
been very good ones. We, as a research center have expressed our reservations about
the evaluation formally, well in advance of the outcome. We raised serious and
factually documentable concerns regarding the competence of a panel member, drew
attention to another member, noting (and anticipating) that people discovering possibly
associations would be screaming ‘bias’ – which is precisely what this blogger
is engaged in. Indeed, I agree with a number of points that this blogger writes
about what has gone wrong and what was wrong in the evaluation process. We are
currently busy with a number of other universities trying to engage the FCT in
reconsidering other initiatives that – in our view- are highly questionable.
I should note that the expression of my concerns are not due to ‘sour grapes’, that is not
as a member of a research center that has been evaluated as ‘fair’ in the last
round and consequently never entered the second round. If such were the case, then
the blogger to this conversation would rightly not believe in my credibility
and suggest that I am shouting ‘sour grapes’.
Let me conclude by clarifying what I mean by ‘vacuous exchange’. Personalized attacks
do not advance anything aside from creating considerable negative energy. It is
in collaborative efforts to advance the quality of science and science policy
that progress can be made and not by slinging mud on each other. It is perhaps
the nature of blogs that creates and enhances such negativity and venom. On
such occasions it makes much more sense to meet and talk constructively rather
than engage in what I think is ‘vacuous exchange’.
@ESF evaluation observer
One correction and one clarification:
“You also got a
value per capita that is larger than the whole funding of about half
the centres under evaluation, including one classified as Very Good.”
This should read “…larger than the whole funding of 12 centres who made it to the second stage, including one with Very Good, and is comparable to the funding of 7 other centres with Very Good and a large number of thoe who stayed behind in the first stage.”
The clarification is that the 6% of units who got more than 50% of the funding refers to 6% of those in the second stage, where most of the funding was given.
@G R Semin
“Clarity in
writing helps clearing confusion – admittedly the original text is
not an example of clarity.“
According to the
Collins English Dictionary, “admittedly” means “wittingly
conceded” which does sound a bit odd here, since you are not the
author of the original text, and, as far as I can tell, the author
himself has not yet stated anything in that direction. What actually
happened was that you misinterpreted the text and then proceeded to
classify your own misinterpretation as being “completely wrong”,
using this to try and discredit other parts of the text – could
this also be classified as an association fallacy or does it have
another name?
But there is a much
more important point where we disagree.
As far as I’m
concerned, the FCT-ESF evaluation is a scam, where what was announced
as being an independent evaluation in absolute terms turned out to be
a biased exercise with quotas being used under the table, where the
evaluation rules were broken at several different points and where, at least
in some areas, there was very little, if any, correlation between
FCT’s own bibliometric indicators and the actual outcome of the
evaluation.
In this setting, I
would say that anyone who brings up the “facts regarding the
quality of a center” is either fooling himself or trying to fool
others.
At best, the results
of this exercise reflect the incompetence with which the whole
process was designed. Just to mention one point, funding was given
away as a percentage of what units had asked for. Clearly this does
not a stable process make, at least not when the total amount
available for funding is finite. In fact it causes serious problems
which have to be solved a posteriori, as they were in this case by
resorting at the last minute to the elimination of the basic funding
and the magical appearance of odd percentages for the strategic
funding for each grade.
When units are
ordered by funding per capita it is not just that your centre got the
highest value. It actually did so by a difference of nearly 50% more
than the funding per capita of the second centre. You also got a
value per capita that is larger than the whole funding of about half
the centres under evaluation, including one classified as Very Good.
In my view, nothing
short of having argued (very convincingly) that you would eliminate
poverty or find a cure for malaria would justify such an outcome…
But there is more to
it than this. Do you really want to discuss the facts regarding the
evaluation of your centre? Do you want to publish online for all the
world to see the first stage reports of the two external referees (those who were
supposed to be the real experts on what was under evaluation),
including the one which gave you your lowest mark?
Other centres had
their lowest mark higher than yours and a similar average but did not
make it to the second stage simply because that low mark was awarded
by the internal referee. What is the difference? This referee was the
gate keeper, the one in charge of making sure that quotas were
followed.
Add to this the fact
that one of the people in charge of the evaluation of your centre had
belonged to the same Standing Committee of Social Sciences which you
had chaired at ESF and it can be no surprise that your centre stands
out like a sore thumb. While other units were being evaluated by
people who were not even from relevant scientific areas, you had a
friendly voice in the panel.
You may call this
whatever you want, but there were centres in circumstances similar to
yours who did not even make it to the second stage. You may think it
unfair that only yours gets to be mentioned. This is because your
centre is emblematic of many of the things which are wrong with the
process. But let’s get this clear: the problem is not your centre,
the problem is the evaluation.
In any case, so that
you don’t feel singled out, here is another example: there is a unit
with precisely the same average of the three marks at the end of the
first stage as your centre, also with one low mark (though not as low
as yours), who made it to get the third largest funding as a whole
(and the 17th largest funding per capita). What is the
“association fallacy” in this case: it is the unit of one of the
vice-presidents of FCT. Lest we forget what is at stake here, the
problem with all this is that there are other units with equal or
even higher marks who did not make it to the second stage, and the
telling sign is whether the lowest mark was given by the internal
referee (not an expert, but with decision power) or by an external
referee (an expert, but with no decision power).
For most of the
units this evaluation was a lottery where 6% of those under
evaluation received more than 50% of the whole funding. Although I
understand that it is not to be expected of the centres who got a
winning ticket to actually come out in support of the annulment of
what is, in my view, nothing short of a fraud, at least I don’t
expect them to pretend that there was any fairness or quality
judgement in the process.
It is a sad day when
the scientific community does not stand up as a whole in the face of
what is an incompetent and biased evaluation exercise, to say the least.
@ESF evaluation observer Clarity in writing helps clearing confusion – admittedly the
original text is not an example of clarity.
Second, there are at least two possible ways in which one
can interpret a center getting the highest funding per capita (but no way near any
of the highest funding). One is interpreting it with a negative intention (and
thus resorting to the use of an association fallacy – again) another is
checking the facts regarding the quality of a center – it is a sad day when a
colleague makes his general case by continuing to use an ad hoc interpretation and singles out a center.
@Semin
“There are 11 units classified as ‘exceptional,’ which is
the highest possible grade. Of the five units with the highest
funding awarded, only one was classified as exceptional. The unit
with the highest funding per capita is in the humanities and is
headed by a former chair of an ESF scientific committee.”
These are three different categories: those with the highest
classification, those with the highest funding, and those with the
highest funding per capita. This is one of the problems: the three
things are poorly correlated.
Paraphrasing you, it is always good to make sure one has
understood a text correctly, before making a judgement.
As for the rest, you forgot to mention
D. one of the panel members who evaluated your centre and, I
believe, was actually one of those who was in the visit, was also in
the same Standing Committee of Social Sciences that you chaired.
My attention has been drawn to the above article written by Carlos Fiolhais. I am aware that there is considerable controversy about the FCT driven ESF evaluation of research centers in Portugal. I am writing to express my disappointment with the inaccuracies and implied innuendos that my colleague has in his piece.
First, it is indeed accurate that 11 centers were graded as exceptional in the research center evaluation. Unfortunately, what follows after this accurate observation is completely wrong.
Let me quote: “Of the five units with the highest funding awarded, only one was classified as exceptional. The unit with the highest funding per capita is in the humanities and is headed by a former chair of an ESF scientific committee.”
The unit implied (highest per capita funding) was NOT classified as ‘exceptional’ but ‘excellent’ and was not among the top 11, because it fell in the one lower grading. Its always good to get one’s facts correct.
Second, I take great offense in the explicitly stated bias in the evaluation because I was the Chair of the ESF Standing Committee of Social Sciences our center must have been favorably graded. My Chairmanship ended in April 2007!
This is an example of a common association fallacy that many people make – but not necessarily scientists – let me illustrate:
A. Dig out two facts. (a) Previous ESF Chair, and (b) highest funding per capita.
B. Implicitly associate them (ESF does the evaluation – therefore);
C. Generate an explanation that ad hoc ties the two together = Biased Evaluation.
It would be nice if the author of this piece would express an apology for insinuating such biases.
Kind regards,
Gün Semin
Dr Gün Semin
Research Professor, Psychology
Director, William James Center for Research
Read Amaya Moro Martin’s reaction to the latest news from Portugal in EuroScientist
http://www.euroscientist.com/bitterness-cannot-be-avoided-but-should-not-turn-into-rejection/