Miguel Seabra-Credit FCT-

FCT head resigns, amid Portuguese research community survival plea

Portugal not yet cleared of the ESF evaluation fallout, despite FCT president’s resignation

Until a few weeks ago, Miguel Seabra was president of the Portuguese Foundation for Science and Technology (FCT). He resigned on 7th April 2015, invoking personal reasons, as rumours about him having kept his position at Imperial College while being at FCT were beginning to circulate. Maria Arménia Carrondo, a researcher from the same laboratory as Seabra has just been appointed by the government as his successor. Portuguese scientists are waiting to see whether she will maintain the status quo or attempt to find constructive solutions. Above all, the Portuguese science community is now calling for increase in transparency and integrity.

In any event, Seabra’s position at the helm of the Portuguese funding agency was becoming untenable. He had been under tremendous pressure following the discredit shed on the FCT by a controversial evaluation of Portugal’s research units, outsourced to the European Science Foundation (ESF). The move was designed to select units that would need to be closed as part of a national austerity drive and due a reduction of European funds that may be used for research. A big question mark has been hanging over the quality of the evaluation since the Portuguese science community started to realise that, in many ways, those bound to survive the cut were not to do so on scientific merit.

Before Seabra resigned, an unexpected 40% cut on the funding of top ranked research units was announced. This follows the planned elimination of 50% of all research units in the country, made public in June 2014. In addition, FCT also announced further restrictions on doctoral and post-doctoral grants. This move will push many researchers into either unemployment or towards emigration, due to lack of alternatives.

Flawed evaluation

As the questionable conditions of the FCT evaluation emerged, the disastrous situation for the Portuguese research community became increasingly obvious to international observers. As it turns out, Spanish astrophysicist Amaya Moro Martin was right. In October 2014, she wrote an opinion piece in Nature, regarding science policy in Europe. She included a reference to Portugal saying: “Already reeling from budget cuts of 50% for universities and research centres, Portugal may now have to close half of its research units because of a flawed evaluation process supported by the ESF.”

Shortly after that, the head of science support of ESF, Jean-Claude Worms, wrote an email to Amaya, threatening her with a lawsuit. ESF backed off on this ridiculous intention a week later, perhaps realising that victimising a young researcher was not a good move. The ESF wikipedia entry sums up the impact the move has had on ESF’s reputation in the following terms: “ESF was very much discredited after a controversial evaluation of the Portuguese research centres.”

To say that the evaluation process is flawed is an understatement. It was a scam setup to justify the closure of half of the research units in the country and concentrate the major financial resources on a small number of units; incidentally, the units related to the current science policy makers in Portugal were awarded generous funding.

Some of the flaws have been identified by members of the Portuguese science community—with details described here and here. They include changing the rules when the process was well underway. And they also involve the use of incorrect bibliometric data. The Council of Rectors of Portuguese Universities (CRUP), in November 2014, protested with strong words, qualifying the evaluation as ‘a total failure.’ Three of the more outrageous issues identified, include the use of hidden quotas, an inadequate evaluation panel and the arbitrary allocation of funding.

Hidden quotas

But let’s get further into the details of what went wrong. The evaluation was carried out in two stages. Only the units proceeding to stage two were eligible for the large part of the funding available. The others will have no funding at all or funding at such a low level that it will not allow them to pursue any activity. In short, units that are not proceeding to stage two are condemned to disappear.

What nobody knew at first is that half of the research units were condemned even before the evaluation had started. After legal pressure FCT made public the contract with ESF. It then became obvious that ESF had been explicitly instructed to a priori exclude 50% of the units from stage two. The work plan for the panels spells it out in the following terms: “Stage 1 evaluation will result in a short list of half of the research units that will be selected to proceed to stage 2.” ESF complied. To this day FCT stills denies the existence of quotas, making their own very peculiar interpretation of what is written in the contract (please check for yourself). Stage one did not include site visits, as is mandatory by law. Sitting at their desks, ESF evaluators simply killed half of the Portuguese research units at a stroke of the keyboard

Light evaluation panel

However, this was not, by far, the only issue. When it comes to the adequacy of the review panels, the operation details of the evaluation speak for themselves. ESF set up a very small number of panels. For example, in the 2007 international evaluation 15 mathematicians, six physicists and seven chemists were gathered in three panels to analyse the units of the corresponding disciplines.

In 2014, the ESF formed a single panel for the three disciplines, with only 11 members (one engineer, three physicists, four chemists and three mathematicians). This panel, which decided to shut down several condensed matter physics labs in the country, had a single condensed matter physicist. In stage two, a team of evaluators visiting an architecture unit did not include a single architect and there were several other examples of clear mismatches between panels and the units they were supposed to evaluate.

Arbitrary funding

Furthermore, the most recent FCT funding allocation does not display a clear correlation with the ESF assessment, considering comparable unit sizes and laboratory intensity. There are 11 units classified as ‘exceptional,’ which is the highest possible grade. Of the five units with the highest funding awarded, only one was classified as exceptional. The unit with the highest funding per capita is in the humanities and is headed by a former chair of an ESF scientific committee.

Get high quality articles on Science policy, innovation and society delivered directly to your inbox.

We respect your privacy.

So FCT asked ESF to make an assessment and, in the end, “modulated”—this is the actual FCT word —the output to award funds in an almost arbitrary way. This means that FCT called ESF as external institution to enforce predetermined funding intentions, regardless of scientific merit. ESF, who was paid a consultancy fee, was only too happy to oblige.

Out of the 322 research units being evaluated, 178 made it to stage two. Meanwhile, the majority of those, which did not make the grade, have appealed the decision, first to FCT, and then by demanding a new evaluation process. The latter is to be carried out by a different panel according to FCT’s own regulations. Four lawsuits are already underway. Out of the 178 “lucky ones,” 123 filed complaints regarding the second stage results, failing to see the logic of the process. As a final despicable move, FCT gave a three-day deadline for units to accept whatever funding befell them.

Portuguese scientists clearly have reasons to be concerned, but maybe they should not be the only ones. Miguel Seabra still remains president of Brussels-based research advocacy group Science Europe, whose aim is to influence European research policies. With current European commission winds blowing against fundamental research, having someone with such a track record in science management cannot be a good omen.

Carlos Fiolhais

Professor of Physics, University of Coimbra , Portugal.

Can the FCT evaluation be rescued by the new management in place?

Your thoughts and opinions are valuable, feel free to use our simple comment section below.

Featured image credit: FCT

EuroScientist is looking for contributors!

If you would like to write guest posts in EuroScientist magazine, send us your suggestions of articles at office@euroscientist.com.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

8 thoughts on “FCT head resigns, amid Portuguese research community survival plea”

  1. @G R Semin 

    Here is a summary of
    the various “vacuous” points which reflect my opinion and which
    were made in the previous posts, plus some other notes:

    1. the latest
    evaluation of Portuguese research units by the tandem FCT-ESF was a
    scam designed to secure funding for some units at the cost of
    eliminating others;

    2. the rules of the
    game were poorly designed and had, to align with the previous point,
    to be changed several times when the evaluation was already in full
    swing; [btw, this alone should be enough for Portuguese courts to
    declare the whole process void]

    3. for many units,
    whether they made it to the second stage or not depended not on the
    opinion of experts but on the decision of a member of the panel who,
    in many cases, was not well equipped to make that decision and was
    under very strong pressure to follow the quotas enforced by FCT; the
    same applies to the decision to attribute the different marks in the
    second stage;

    4. the distribution
    of funding did not follow the rules initially announced by FCT and
    the percentage of the funding corresponding to each of the final
    marks was set after the fact in order to keep within the budget; this
    was completely predictable, since the method of giving a percentage
    of what units ask for is not only idiotic but also impossible to
    follow in practice;

    5. apart from other
    evidence, the effects of the above may be seen from a careful
    analysis of the results; for instance, even before the contract
    between ESF and FCT describing the 50% quota was made public, this
    fact had already been discovered from an analysis of the data
    available at that time;

    6. this analysis
    extends to other general issues such as the fact that the only
    quantitative piece of data which showed any correlation to whether a
    unit made it to the second stage or not was its size; one possible
    interpretation of this is that this part of the evaluation was
    extremely superficial;

    7. the many
    evaluation flaws of the system also transpire in other ways, as is
    the case of some specific units; whether you like it or not, your
    centre’s evaluation reflects the appalling funding strategy enforced
    by FCT;

    8. you may continue
    arguing about whatever fallacies you want until you’re out of breath,
    but the bottom line is the following: for many, if not most, units,
    this evaluation was a lottery;taking that into consideration, under the conditions
    of this evaluation, if people had had a choice, would they have
    preferred to be evaluated by a panel where they knew one person (with
    whom they were on good terms) and where at least one of the other
    members was not fit to evaluate their unit, or by a panel where they
    did not know anyone and none of the members was from a relevant area
    [as happened to many other units]?

    9. during the period
    when units may protest, all the relevant information concerning the
    evaluation, including the reports, may, by Portuguese law, be made
    available to the other units upon request.

  2. @ESF evaluation observer
    Gün R Semin

    I am not sure with whom I am having this virtual conversation
    since the author of the blog does not sign with a name. Whoever the participant
    to this exchange may be – this is turning out to be a ‘vacuous
    exchange’. Let me continue with by adding some examples by my anonymous
    partner to this exchange. First, let me provide an example provided by a
    colleague of mine.

    A burning cigarette was carelessly discarded.
    Several acres of virgin forest were destroyed.

    You could let these two statements stand as two
    unrelated utterances. “But that’s not what you did, right?”
    What is the difference between this example and
    the examples this blogger uses to insinuate bias?
    Here another example of a vacuous point about the use of English
    phrases, such as ‘admittedly’, and I quote: “According to the Collins English Dictionary,
    “admittedly” means “wittingly conceded”
    which does sound a bit odd here, since you are not the author of the original
    text, and, as far as I can tell, the author himself has not yet stated anything
    in that direction. What actually happened was that you misinterpreted the text
    and then proceeded to classify your own misinterpretation as being “completely
    wrong”, using this to try and discredit other parts of the text – could this
    also be classified as an association fallacy or does it have another name?”

    This starts with an unfortunate and self-centered definition of ‘admittedly’ illustrative
    of the type of rhetorical games straddling this vacuous dialogue [more about why
    this dialogue is vacuous later].Let us start by rewinding. Admittedly is a term “… used to introduce a concession or
    recognition that something is true or is the case: admittedly, the salary
    was not wonderful, but the duties were light | this is admittedly an
    extreme case.” – which translates to – and I quote my text again “admittedly,
    the original text is not an example of clarity” meaning an exercise in
    confusion, or meaning the ‘text is not an example of clarity to a reader’ (not
    sure where the author of the blog comes into this except that the writing is not clear).
    Do we want to continue quibbling? It’s a waste of time. The only contentious issue is
    unjustified insinuations – but obviously, admissions are not on the agenda!
    Admittedly, the author of the blog has insider information about
    the inner workings of the FCT, which I do not have. It interesting to think
    about how this privileged information was accessed. One-way is via the FCT
    (very unlikely) and the second is sad even to consider. Whatever, if the blogger
    were really well informed when attacking the center, then s/he would have to be
    better informed about specific details during the evaluation process.

    Admittedly, the FCT evaluation process does not win first
    prize in a beauty contest, although the original intentions of the FCT may have
    been very good ones. We, as a research center have expressed our reservations about
    the evaluation formally, well in advance of the outcome. We raised serious and
    factually documentable concerns regarding the competence of a panel member, drew
    attention to another member, noting (and anticipating) that people discovering possibly
    associations would be screaming ‘bias’ – which is precisely what this blogger
    is engaged in. Indeed, I agree with a number of points that this blogger writes
    about what has gone wrong and what was wrong in the evaluation process. We are
    currently busy with a number of other universities trying to engage the FCT in
    reconsidering other initiatives that – in our view- are highly questionable.

    I should note that the expression of my concerns are not due to ‘sour grapes’, that is not
    as a member of a research center that has been evaluated as ‘fair’ in the last
    round and consequently never entered the second round. If such were the case, then
    the blogger to this conversation would rightly not believe in my credibility
    and suggest that I am shouting ‘sour grapes’.

    Let me conclude by clarifying what I mean by ‘vacuous exchange’. Personalized attacks
    do not advance anything aside from creating considerable negative energy. It is
    in collaborative efforts to advance the quality of science and science policy
    that progress can be made and not by slinging mud on each other. It is perhaps
    the nature of blogs that creates and enhances such negativity and venom. On
    such occasions it makes much more sense to meet and talk constructively rather
    than engage in what I think is ‘vacuous exchange’.

  3. @ESF evaluation observer 
    One correction and one clarification:
    “You also got a
    value per capita that is larger than the whole funding of about half
    the centres under evaluation, including one classified as Very Good.”
    This should read “…larger than the whole funding of 12 centres who made it to the second stage, including one with Very Good, and is comparable to the funding of 7 other centres with Very Good and a large number of thoe who stayed behind in the first stage.”

    The clarification is that the 6% of units who got more than 50% of the funding refers to 6% of those in the second stage, where most of the funding was given.

  4. @G R Semin 

    “Clarity in
    writing helps clearing confusion – admittedly the original text is
    not an example of clarity.“

    According to the
    Collins English Dictionary, “admittedly” means “wittingly
    conceded” which does sound a bit odd here, since you are not the
    author of the original text, and, as far as I can tell, the author
    himself has not yet stated anything in that direction. What actually
    happened was that you misinterpreted the text and then proceeded to
    classify your own misinterpretation as being “completely wrong”,
    using this to try and discredit other parts of the text – could
    this also be classified as an association fallacy or does it have
    another name?

    But there is a much
    more important point where we disagree.

    As far as I’m
    concerned, the FCT-ESF evaluation is a scam, where what was announced
    as being an independent evaluation in absolute terms turned out to be
    a biased exercise with quotas being used under the table, where the
    evaluation rules were broken at several different points and where, at least
    in some areas, there was very little, if any, correlation between
    FCT’s own bibliometric indicators and the actual outcome of the

    In this setting, I
    would say that anyone who brings up the “facts regarding the
    quality of a center” is either fooling himself or trying to fool

    At best, the results
    of this exercise reflect the incompetence with which the whole
    process was designed. Just to mention one point, funding was given
    away as a percentage of what units had asked for. Clearly this does
    not a stable process make, at least not when the total amount
    available for funding is finite. In fact it causes serious problems
    which have to be solved a posteriori, as they were in this case by
    resorting at the last minute to the elimination of the basic funding
    and the magical appearance of odd percentages for the strategic
    funding for each grade.

    When units are
    ordered by funding per capita it is not just that your centre got the
    highest value. It actually did so by a difference of nearly 50% more
    than the funding per capita of the second centre. You also got a
    value per capita that is larger than the whole funding of about half
    the centres under evaluation, including one classified as Very Good.

    In my view, nothing
    short of having argued (very convincingly) that you would eliminate
    poverty or find a cure for malaria would justify such an outcome…

    But there is more to
    it than this. Do you really want to discuss the facts regarding the
    evaluation of your centre? Do you want to publish online for all the
    world to see the first stage reports of the two external referees (those who were
    supposed to be the real experts on what was under evaluation),
    including the one which gave you your lowest mark?

    Other centres had
    their lowest mark higher than yours and a similar average but did not
    make it to the second stage simply because that low mark was awarded
    by the internal referee. What is the difference? This referee was the
    gate keeper, the one in charge of making sure that quotas were

    Add to this the fact
    that one of the people in charge of the evaluation of your centre had
    belonged to the same Standing Committee of Social Sciences which you
    had chaired at ESF and it can be no surprise that your centre stands
    out like a sore thumb. While other units were being evaluated by
    people who were not even from relevant scientific areas, you had a
    friendly voice in the panel.

    You may call this
    whatever you want, but there were centres in circumstances similar to
    yours who did not even make it to the second stage. You may think it
    unfair that only yours gets to be mentioned. This is because your
    centre is emblematic of many of the things which are wrong with the
    process. But let’s get this clear: the problem is not your centre,
    the problem is the evaluation.

    In any case, so that
    you don’t feel singled out, here is another example: there is a unit
    with precisely the same average of the three marks at the end of the
    first stage as your centre, also with one low mark (though not as low
    as yours), who made it to get the third largest funding as a whole
    (and the 17th largest funding per capita). What is the
    “association fallacy” in this case: it is the unit of one of the
    vice-presidents of FCT. Lest we forget what is at stake here, the
    problem with all this is that there are other units with equal or
    even higher marks who did not make it to the second stage, and the
    telling sign is whether the lowest mark was given by the internal
    referee (not an expert, but with decision power) or by an external
    referee (an expert, but with no decision power).

    For most of the
    units this evaluation was a lottery where 6% of those under
    evaluation received more than 50% of the whole funding. Although I
    understand that it is not to be expected of the centres who got a
    winning ticket to actually come out in support of the annulment of
    what is, in my view, nothing short of a fraud, at least I don’t
    expect them to pretend that there was any fairness or quality
    judgement in the process.

    It is a sad day when
    the scientific community does not stand up as a whole in the face of
    what is an incompetent and biased evaluation exercise, to say the least.

  5. @ESF evaluation observer Clarity in writing helps clearing confusion – admittedly the
    original text is not an example of clarity.
    Second, there are at least two possible ways in which one
    can interpret a center getting the highest funding per capita (but no way near any
    of the highest funding). One is interpreting it with a negative intention (and
    thus resorting to the use of an association fallacy – again) another is
    checking the facts regarding the quality of a center – it is a sad day when a
    colleague makes his general case by continuing to use an ad hoc interpretation and singles out a center.

  6. @Semin 

    “There are 11 units classified as ‘exceptional,’ which is
    the highest possible grade. Of the five units with the highest
    funding awarded, only one was classified as exceptional. The unit
    with the highest funding per capita is in the humanities and is
    headed by a former chair of an ESF scientific committee.”

    These are three different categories:  those with the highest
    classification, those with the highest funding, and those with the
    highest funding per capita. This is one of the problems: the three
    things are poorly correlated.
    Paraphrasing you, it is always good to make sure one has
    understood a text correctly, before making a judgement.
    As for the rest, you forgot to mention
    D. one of the panel members who evaluated your centre and, I
    believe, was actually one of those who was in the visit, was also in
    the same Standing Committee of Social Sciences that you chaired.

  7. My attention has been drawn to the above article written by Carlos Fiolhais. I am aware that there is considerable controversy about the FCT driven ESF evaluation of research centers in Portugal. I am writing to express my disappointment with the inaccuracies and  implied innuendos that my colleague has in his piece.
    First, it is indeed accurate that 11 centers were graded as exceptional in the research center evaluation.  Unfortunately, what follows after this accurate observation is completely wrong.
    Let me quote: “Of the five units with the highest funding awarded, only one was classified as exceptional. The unit with the highest funding per capita is in the humanities and is headed by a former chair of an ESF scientific committee.”
    The unit implied (highest per capita funding)  was NOT classified as ‘exceptional’ but ‘excellent’ and was not among the top 11, because it fell in the one lower grading. Its always good to get one’s facts correct.
    Second, I take great offense in the explicitly stated bias in the evaluation because I was the Chair of the ESF Standing Committee of Social Sciences our center must have been favorably graded. My Chairmanship ended in April 2007!
    This is an example of a common association fallacy that many people make – but not necessarily scientists – let me illustrate:
    A. Dig out two facts. (a) Previous ESF Chair, and (b) highest funding per capita.
    B. Implicitly associate them (ESF does the evaluation – therefore);
    C. Generate an explanation that ad hoc ties the two together = Biased Evaluation.
    It would be nice if the author of this piece would express an apology for insinuating such biases.
    Kind regards,
    Gün  Semin
    Dr Gün Semin
    Research Professor, Psychology
    Director, William James Center for Research