failure

Casper Albers is chair of Applied Statistics and Data Visualisation at the University of Groningen. With a background and degree in mathematical statistics, he now works at the department of Psychology. You can find out more about him on his website, or by reading his tweets.

1. Hi Casper, thanks for joining How I Fail! Next to your official bio, could you say something more to introduce yourself?

Hi Veronika, thanks for having me on your series! My main line of research involves the analysis of so-called momentary assessment data and other types of longitudinal data in psychology. With the rise of new technologies, such data have become more abundant in psychological research, yet still too little is know about the best ways to build relevant statistical models. Furthermore, I’m interested in statistical communication: how can we make sure that the main message of an advanced statistical model is conveyed correctly to non-technical end-users of the model. Furthermore, I’m a member of the University Council and thus involved in university politics.

I’ve done my undergraduate and PhD in mathematical statistics in Groningen, then did my first postdoc in bioinformatics. After spending four years as postdoc at the Open University in England, I returned to Groningen in 2009 and haven’t left since.

2. You recently wrote a column about being rejected for a prestigious grant (thanks for the shout-out!). Can you share a bit more thoughts about this?

This was an application for the so-called NWO Open Competition scheme. The goal was to obtain money for a PhD-student and postdoc on a project combining my two lines of research: momentary assessment data and communication. It was a ‘pre-proposal’, with a 500 word limit for the main text, and about half the candidates would’ve been invited to write a full proposal. I was part of the other half.

Initially, I felt frustration, of course. One reviewer mentioned a paper (published a week before the deadline) I didn’t cite, and asked why I requested a specific international visit. I did read this paper (but could not cite it due to the limit to the number of allowed citations), and the visit would be to the authors of this paper. This reviewer clearly did not read the proposal clearly. Having put in hours of work myself, that feels very frustrating.

Furthermore, all five reviewers indicated that I needed to elaborate in more detail, which to me seems impossible given the limited word count. Initially, you feel frustrated and wrongly treated. Once the frustration wears off, you realise that the reviewers reviewed multiple proposals. If all five of them claim my proposal lacked detail, then I guess others did succeed in providing more detail and did rightfully ‘win’ at the expense of me. It’s still frustrating of course, but I’ve come to terms with it. I’m now looking for alternative funding possibilities for (part of) the proposal.

3. Are there any other failures you’d like to share that were particularly memorable?

I was a postdoc at the Open University between 2005 and 2009. Near the end of my contract, we applied for a grant with me as named postdoc at a British funder, under the category ‘mathematics for business’. After the deadline, we received a letter stating that out of all proposals in this category, we were ranked first. Shortly thereafter, a second letter followed: due to a large amount of proposals in the medical categories, none of the proposals in our category would be funded. Thus, we had the best proposal, yet still received nothing.

Having participated in a funding scheme that turned out to have no money is a nice story for birthday parties, so I did get something out of it…

4. Next to grant and job rejections, are there any other things that fall under the word failure for you?

I would be hesitant to even call grant and job rejections failures. The word failure indicates that I did something wrong. Grant schemes often have a success rate of about 15%. There’s nothing wrong with belonging to the 85% – especially if you also occasionally belong to the 15%. If the system is such that the system expects you to be part of the lucky 15% every time, then the system is broken. That was also my point in my column, and I think is also yours in this series: failing is normal.

A couple of years ago, after working for this university for nearly 15 years, I got a permanent contract. Last August, I was promoted to professor. So, I’m currently in the more senior part of my career where grants are very helpful, but not essential anymore. That helps tremendously in being able to distance myself from this. When I still had job insecurity, this was much harder, of course.

Instead of failures, I’d like to call grant and job rejections major disappointments. Many other things in academia are as well: paper rejections (especially when the review’s poor or when it involves the first paper of your PhD-student), not meeting deadlines, etcetera. These disappointments are on a continuous scale. I guess how disappointing something bad happening is mainly depends on how unexpected it is and how much of an impact it has on me.

5. Are there are any caveats with sharing failures online?

With just sharing news like “I got rejected for this grant”, I don’t see any caveats. On the contrary, I think it is very good if ECRs get informed that established names also had to struggle through many failures. Having your grant proposal rejected or not being offered a job does not make you unsuitable for academia. Although there’s clear survivor bias in stories like mine. It would be interesting to also hear from people who decided to leave academia because of (too many) grant/job rejections.

But, of course, you can share too much information online. I can totally understand if someone doesn’t want to share the details of her/his rejected proposal online, out of fear of someone scooping your ideas.

6. Now that the grant rejection gave you some time to read all the How I Fail interviews :), what are your favorite take-aways from the series?

I like what Daniel Lakens wrote about mastery and performance goals a lot. I also very much liked various persons, e.g. Melanie Stefan, talking about privilege. For a straight white, male senior-researcher it is easy for me not to realise how privileged I am. It is good to read about the additional struggles that people from other backgrounds have, also in academia. This has certainly shaped the way I look at many academic processes.

7. Who should I ask as the next person I am interviewing for How I Fail?

It would be really interesting to read interviews with people who decided to leave academia because of failures, so to read about the other side of the survivor bias coin. As these people left academia, they also kind of left my ‘bubble’, so I don’t have any names for you.

8. What should I ask the next person I am interviewing for How I Fail?

Whether they have any experiences from the past that they at the time considered a success but now, in hindsight, consider a failure. As an example: massaging your data such that p < .05, making the paper publishable in some ‘good’ journal used to be regarded as a success; and we now know that p-hacking is naughty.

9. Is there anything that you feel like you are currently failing at yourself?

Time management. I always plan to do more in a week than what I can do. This is not a one-off failure as a grant/job rejection, but a source of constant frustration.

10. What do you think your past-self of N-10, N-20, N-30 etc years would think of you now?

My N-30 me would observe that I earn my money by solving puzzles on a computer all day, and playing with LEGO in the evening. I think that would make my N-30 me quite happy. My N-15 me would also be quite happy: I wanted a job in academia, and I got a job in academia.

11. What is the best piece of advice you could give to these past selves?

I’d give my past self a copy of Grays Sports Almanac 1950-2000 and could then fund myself. This works, I’ve seen it in the movies.

More seriously: I’d advise myself to spend less energy jumping through the hoops the system had set up for me, and spend more energy on just doing what I like to do and what I’m good at (fortunately, these two correlate quite well for me). This is fine.

12. What do you think about the current way success/failure are influencing academic research in the Netherlands?

It’s not healthy. Apart from my four years in the UK, I’ve been working at this university since 1996, so that’s about eighteen years. And I’m still in a system where I have to prove myself: if I don’t obtain a grant within seven years (at that time I’ll be fifty and will have literally worked half my life for this university), the university could strip me of my professor title. This puts stress on employees where it is not necessary. If the university can’t decide whether I’m worthy of the title “prof. dr. Albers” after eighteen years, then that says more about the university than about me.

My university is not special in this regard, all Dutch universities are roughly the same in this. Especially when we know that so many things go wrong with grant funding, such as the Matthew effect which has recently been proven to be very prominent at NWO, this is not a very evidence-based/scientific way to run academia. (Note that the university obviously not only selects and promotes based on grant success, and also includes a lot of sensible measures).

13. Do you have any suggestions on how we could change this? What is a concrete step that somebody reading this post could take in this direction tomorrow?

Focus on quality rather than quantity. Success in the Netherlands is mainly measured based on how many papers you have, how many citations you have, your H-index, etc. You can have one excellent paper, or a paper with a lot of societal impact yet few citations. You can also salami slice a semi-decent study into a series of boring papers. If I had to pick which of these versions of you would get a promotion, I would definitely go for the first version.

Furthermore, we should step away from all kinds of quantitative ways of measuring quality and including these ways in assessments. Goodhart’s law – When a measure becomes a target, it ceases to be a good measure – is over forty years old, but still spot on.

Criticising things is, of course, easy. I don’t have an easy solution either. Stepping away from emphasising the importance of all these quantitative, semi-objective measures and just using common sense should be a big step forward.

14. Is this something you discuss with the researchers that you mentor? Is there anything they can do too, to change things?

Yes, we discuss these things. With one of my PhD-students, for instance, we decided to combine two studies we did in one single, hopefully high impact, paper; rather than trying to get both studies published separately, which was the original idea.

But I think that early career researchers cannot do too much: they are at the bottom of the academic food chain, and for them it’s much harder to be heard, especially individually. When they join forces, such as in the Young Academy in the Netherlands, they do reach an audience.

But the main change should come from senior researchers, like me, who are in the position to demand change. With great power comes great responsibility.

Caroline Rowland runs the Language Development Department at the Max Planck Institute for Psycholinguistics in Nijmegen, and is one of five directors at the ESRC LuCiD Centre. She studies how children learn language, or, more concretely: what are the mechanisms in the human brain that give us the capacity to acquire language?  She’s on Twitter as @CaroRowland.

1. Hi Caro, thanks for joining How I Fail! Can you tell us a little bit about yourself?

I’m a Brit who escaped Brexit by miraculously being offered a job in the Netherlands in 2016. Before that I worked in various universities in the UK, except for one year when I worked as a secretary in various offices around London. I didn’t get a good enough undergraduate degree to obtain funding for a PhD so I did it part-time (self-funded) while working as a Research Assistant. I then moved to the University of Liverpool for 16 years before taking a job at the MPI in Nijmegen.

2. Would you like to share some failures with us?

Oh yes, indeed. There are countless examples; failure at exams (failing S-level history at age 18 still smarts), failure at job interviews (I was rejected from teacher training college), failure with grants and journals. Reviewers have told me that my work is meaningless (“Rowland and her colleagues just count things”), that I work too closely with a small group of people (One of my major reservations is the fact that many of the investigators have already been working together very closely), and that I don’t know what I’m doing (even with significant improvements [this paper] would still not meet typical requirements for peer-reviewed quality papers).

But these kinds of comments are quite unusual. Many of the rejections I have received have been kind and constructive. Even annoying comments like “do X”, when you’ve patently done X on page 12, can be useful – they help you discover what points a hasty reader will miss.

3. What does “failure” mean to you? Are there some misconceptions about it?

Actually, I think ‘fear of failure’ is worse than actual failure. By this, I mean the constant niggling worry in the back of your mind that you’re not quite living up to expectations. For me, it’s not just a fear of failing at research. I live with a background low level worry that I’m not a good enough boss, mentor, colleague and, especially, that I’m not a good enough mother. I find this much harder to manage that actual failure. If anyone has a solution …

4. Do you keep track of failures? Why/why not?

No. Absolutely not. I try to learn from it and move on. If it’s a rejected paper, I revise it to make it better and submit it somewhere else. If it’s something more serious (e.g. that I’ve dealt badly with an issue or colleague at work) I try to apologise, learn from it and move on. I also try not to live with regrets; it’s too exhausting.

5. What do you think about sharing failures online?

Well it’s certainly not objective evidence for a recipe for success, but then it’s not meant to be. It’s narrative, and narrative can be cathartic. Misery loves company. I find it much easier to deal with the immediate aftermath of rejection when I can open a bottle of wine and spend a couple of hours reading stories about how other people have failed too.

But it’s also useful. The recent Twitter discussion we had on grant rejections prompted me to start talking to our researchers and students about how we can better support them in grant writing.

Thread: Grant funding has raised its ugly head again. Dozens of amazing researchers have failed to secure funding. And are despairing – what's the point of all that work when success rates are low & mainly a lottery? Why bother when your fate is in the hands of lady luck?

— Dr Caro Rowland (@CaroRowland) June 4, 2018

6. Have you experienced differences in attitudes towards failure throughout your career?

Of course. For example, it is much easier for me to accept my own failures now I’m older. In my twenties, I found it really hard to acknowledge being wrong. This is partly due to experience – when you’ve spent 18 years being regularly rejected, you learn to deal with it or you spend your life angry. But in my case, I also had a perfectionist PhD supervisor, so I didn’t have a choice. This was difficult to deal with at the time, particularly when a manuscript came back for the 9th time with the comment “I still don’t like this”. But it was also the best training for dealing with rejection.

7. Do you think different groups of people experience failure differently?

It’s much easier to deal with when you’re older (see 6 above). It’s probably also easier if you’re privileged. This tweet on the #ShareYourRejections thread has got me thinking about that a lot ….

So, yea this confirmed for me, I get #ShareYourRejections point noones perfect we can overcome blah blah but… so far looks like good opportunity for some introspection & observations re white privilege? Even just allowed comfiness w sharing failures & praise we receive for it? https://t.co/TfPjdOPQ4D

— The Gonzo Scientist, PhD (@GonzoScientist1) August 17, 2018

8. Is there anything that you feel like you are currently failing at yourself?

Pretty much everything (see answer to 3 above). But my most spectacular current failure is my failure to learn Dutch. I *have* to learn Dutch – I’ll be living in the Netherlands for many years. And I’m a language acquisition expert, so I know that it’s going to be hard; it’s much more difficult to learn a language at my age than it would have been 20 years ago. But I am still constantly disappointed by my failure to be able to hold a conversation in Dutch. I know quite a lot of words already, but retrieving them at speed, in the right order, during a conversation, is almost beyond me.

9. If you could reach all academics with a single message, what would that message be?

Academia is hard, sometimes brutal. But most jobs are hard. So this isn’t a good reason to quit. If I ever start to feel that the problems of academia outweigh the benefits, I imagine myself in a different job; as a teacher, as a lawyer, or as a businesswoman. I imagine how I would feel every Monday morning, faced with a week of standing in front of a class, or week in an office, or a week in a courtroom. Would I feel more or less content? If the answer is ever “more content”, I will start looking round at my options.

10. If you were in a position to change how funding/hiring/etc decisions were made, what would you do?

This one’s easy. I’d allow the grant or hiring committee ample time to read the applicants’ papers, rather than relying on statistics like H indices and citation counts. This is how the Max Planck Society hires, and it’s been a wonderful experience to be given the freedom, and the time, to read what applicants have published.

11. What can this series do better to improve things for everyone?

I think it’s already doing a great job of raising awareness. It certainly helps me in my “misery loves company” moments. It’s also an opportunity to link to resources that might help people cope with the pressure of rejection (e.g. the Samaritans in the UK do a great job of helping people deal with catastrophic failure. But then I’m a psychologist – counselling is always my default option.

12. What is the best piece of advice you could give to your past self?

I’ve already tweeted it. But it’s not about being a scientist – it’s about being a working parent. I really wish I’d known this when I was younger.

I worried about being a working mum for 20 years, feeling guilt every day for doing both jobs less than perfectly. Then to my surprise discovered my daughters saw me as a role model; proving you can be mum *&* have successful career! Eternally grateful to them for telling me this https://t.co/tMUYpMSJmB

— Dr Caro Rowland (@CaroRowland) June 6, 2018

Greg Wilson has worked for 35 years in both industry and academia, and is the author or editor of several books on computing and two for children. He is best known as the co-founder of Software Carpentry, a non-profit organization that teaches basic computing skills to researchers. You can find out more about him on his website.

1. Hi Greg, thanks for joining How I Fail! Can you tell us a little bit about yourself?

Hi Veronika; thanks for having me. I’m Canadian, I wrote my first program on punch cards in 1980, and my family doesn’t let me use power tools (I’m kind of clumsy).

2. What’s on your list of failures?

Oh, I have a loooong list. Off the top of my head, I have half a dozen unfinished book projects that I only call “unfinished” because I’m not honest enough to call them “abandoned”, all three of the open source software projects I started myself fizzled out years ago, and when I teach software engineering, I use stories from my first startup as examples of what not to do.

3. Was there any particular event that made you decide to join this series?

I organized a workshop earlier this year for programmers who want to learn about community organization because I think it’s pretty clear that decisions about tech are being made by people who either don’t understand it, or don’t care about the collateral damage caused by their greed. About two dozen people showed up, and there was a lot of lively discussion, but there wasn’t much follow-up from participants. I’ve been trying to figure out why not, and I’ve been drawing on my past failures to give myself a vantage point.

4. What does “failure” mean to you?

I think of “weak failure” as “I put in a lot of work and nobody got much out of it”. I think of “strong failure” as “I put in a lot of work and did harm that could have been avoided”. By that standard, my thesis was a weak failure, and things like Twitter and Facebook are strong failures. The fact that many people think they’re successful is, to me, the biggest misconception people have about what failure actually is.

5. Do you keep track of failures?

Absolutely: sometimes, late at night, I’ll put on some Johnny Cash and line my failures up side by side and count ’em one by one. Then I’ll have a look at this cartoon and remind myself that I’m an idiot sometimes and then have some hot chocolate and go to sleep.

6. What do you think about sharing failures online?

Yeah, I used to be one of those guys who said “fail early, fail often, fail loudly”. I also used to tell people that their Stack Overflow profile was their real resume, and that if they weren’t contributing to open source projects they weren’t “real” programmers, so I guess by my own definition, my younger self was a strong failure.

What I’m trying to do now with pieces like this is help people like my younger self understand why these positions are wrong by saying, “This is who I used to be/what I used to believe, and here’s why it was harmful, and here’s a better way to think/speak/act.” I also find that I can only really know what I think about something when I hear myself say it out loud, or when I write it down, so sharing my failures helps me figure out what if anything I learned from them.

7. Have you experienced differences in attitudes towards failure throughout your career?

Yes, but they’ve mostly been self-inflicted: I dwell on my failures much more often than I take pride in my successes. I don’t think that’s healthy, but then, I just ate a completely unnecessary butter tart, so…

8. How do you think the changing landscape in computer science [men/women, number of people in general] affects our relationships towards failure?

There’s a lot more discussion in tech companies and open source communities than there used to be about inclusivity, mental health, and unearned/unrecognized privilege. Clearer and more critical analysis of things like “Stack Overflow is your resume” and “if you’re not working weekends, you’re not really committed to success” is part of that.

9. What can we learn from how failures are dealt with in software engineering?

I think we can learn that people are able to rationalize and explain away almost anything 🙂

10. Is there anything that you feel like you are currently failing at yourself?

Yup. I have so many projects on the go that I’m not making much headway on any of them; I keep telling myself to set some aside, but that’s working about as well as swearing off butter tarts.

11. If you could reach any group of people with a single message, what would that message be?

I would tell everyone – everyone – to get out and vote.

12. I understand you’ve read all the previous interview of the series – can you share (i) something that was a new insight for you and (ii) something you disagreed with?

The thing I remember most clearly is Felienne Hermans’ comment that “Felienne of 2013 would not believe me, I’m sure of that”. I’m equally sure that Greg Wilson of 2010 would probably not believe how much and how little the Greg Wilson of 2018 has accomplished. I honestly can’t recall anything that I strongly disagreed with – I spent most of my time nodding and bookmarking as I read the other interviews.

13. What can this series do better to improve things for everyone?

Be more widely read 🙂

14. What is the best piece of advice you could give to your past self?

Be kind; all else is details.

Ian Goodfellow is a staff research scientist on the Google Brain team, where he leads a team of researchers studying adversarial techniques in AI. He was included in MIT Technology Review’s “35 under 35” as the inventor of generative adversarial networks. He is the lead author of the MIT Press textbook Deep Learning. You can find out more about Ian on his website and on Twitter.

1. Hi Ian – thanks for joining How I Fail! I have to admit, I almost failed by being too scared to invite you, so I’m very excited you agreed. Can you tell us a little bit about yourself?

Thanks for inviting me!

I’m an AI researcher at Google and I lead a team of other researchers. We’re working to understand failures of AI better so that we can establish clear engineering principles for responsible AI development. I spend most of my own time studying how to make AI secure—for example, how to make sure that malicious attackers can’t fool AI-based systems into doing what the attacker wants instead of what the designers intended.

2. On Twitter you posted a list of rejections – can you elaborate on these a bit, for an “unofficial bio” of sorts?

My colleague Moritz Hardt tweeted to remind everyone that it was the “time of the year to keep in mind that the typical start of a successful academic career is getting rejected from a bunch of good grad schools.”

I replied with a list of grad schools and fellowship stipends that I was rejected from.

That one story probably is not much of a bio. Here is more of a bio mentioning some failures along the way:

• As an undergrad at Stanford, I struggled in biology and chemistry classes while preparing for a career in neuroscience. I got OK grades, but I didn’t think I was doing well enough to become a professor.
• After I changed my focus to computer science, I applied for several internships as an undergrad. Notably, Google rejected me from an internship.
• I once applied for a summer internship with a Stanford professor. My transcript was included in my application. He replied “Why do you have an A in my class?” It turned out I wasn’t actually meant to have an A in his class. I thought there had been a generous curve, but there had only been a computer glitch. The result of my internship application was that Stanford downgraded my transcript.
• Other large tech companies gave me internship offers, but not to work in machine learning or computer vision. On my CV, you don’t see these failures, just the eventual successes (I’m very grateful that Willow Garage and Stanford’s CURIS program gave me the chance to work on vision for robotics during summer internships)
• In both my masters and my PhD, I spent most of my time without an outside fellowship. This meant I had to work as a teaching assistant or work on specific paying grants rather than focusing primarily on my research interests. I continually applied for fellowship’s like Quebec’s PBEEE. I spent 2009-2013 trying and failing to get open-ended funding until in 2013 I Google gave me the first PhD Fellowship in Deep Learning.
• Vision conferences like ECCV rejected most of the papers I wrote before my PhD. I did a lot of work on perception for robots that never saw the light of day.

3. What factors do you think helped you overcome these setbacks?

In high school, I spent three years on my school’s debate team, coached by two really great teachers: Kerry Koda and Thomas King.

I’ve been surprised how many different aspects of a career in science my debate experience was able to help me with. In terms of overcoming setbacks, debate is useful because debaters all learn how to deal emotionally with failure. Every debate round has a winner and a loser. No one is so good that they always win. If you stick with debate for very long, you quickly get used to the idea of losing a round and then immediately going to another classroom and doing another round, failing to place at a tournament and then immediately going to another tournament the next weekend. You learn not to ruminate and beat yourself up. Also, your expectations get adjusted a lot. You get used to having a constant stream of both failures and successes.

4. There have been some responses to CV of Failures being a humblebrag or a sign of privilege – what would your answer to that be?

When I tweeted about this before, people didn’t react that way. A lot of people thanked me for sharing my rejections. I can definitely understand why people would see this as a humblebrag, but I think most people also understand that I’m doing this to help other people escape impostor syndrome.

5. On the opposite end, do you think that with failure being common, people might decide not to share their successes?

No, we’re all basically forced to share our successes, either for performance review at work, or in grant applications, etc. When submitting papers to conferences and journals, everyone is very incentivized to show the upsides of their paper and try to sweep the downsides under the rug. I personally fight this incentive as much as possible but I don’t imagine it going away in general anytime soon.

6. So far we are talking about successes and failures as things with discrete decisions, like positions and publications. Are there also other things that we can fail at?

I actually think that most discrete points of failure (acceptance / rejection to a specific grad school, or acceptance / rejection of a paper submitted to a conference, etc.) do not matter all that much.

I tweeted about being rejected from a lot of graduate schools, but that was fine, because I was accepted to a lot of others.

For example, in 2009, the largest obstacle for me was not that I had been rejected from some top schools, like MIT and Carnegie Mellon. The largest obstacle for me was that I wasn’t sure I would be able to do the research I wanted to at other top schools where I was accepted, like Stanford and Berkeley. It wasn’t clear who my advisor would be at either school (because there is a rotation program for new PhD students; the advisor is not assigned at the time of your admission offer), and relatively few potential advisors at these schools were supportive of deep learning research. I overcame this obstacle by going to Université de Montréal, with Yoshua Bengio locked in as my advisor ahead of time.

Probably the failure I consider the biggest is that I spent most of my PhD trying to solve supervised learning for computer vision using unsupervised feature learning methods, and was caught totally off guard when Alex, Ilya, and Geoff won the ImageNet contest with purely supervised methods. I think that in general wasting time writing papers that turn out to be dead ends is the main way that I fail in my own eyes. Especially now that it’s normal to post papers on arxiv.org, I consider my work a success if it influences other researchers, even if it gets rejected from a conference, and I consider my work a failure if it has little influence, even if it gets accepted to a conference.

7. Is there anything that you feel like you are currently failing at, or you are hesitant about in the future?

For my own point of view: I’ve been working on understanding why neural networks are easily confused by small perturbations to their input (both through my own direct research effort and working to promote interest in the topic to get other researchers to solve it) for nearly 4 years, and still no one knows how to make a model with high accuracy in this setting.

From the point of view of traditional metrics of career success: The reviews of my papers submitted to ICML this year were particularly brutal and I expect most of them to be rejected.

Another thing I think is definitely worth mentioning: The way that I work, I rapidly try out several ideas to see if they show promise, and discard most of them. On a good day when I get a lot of time at my desk, I might code up 3-5 ideas and decide that none of them work. The time investment per idea is small but I can try out a large amount of relatively different ideas. From this point of view, failure of specific ideas is just a constant part of my workflow.

8. When we talk about successful researchers, what do you think about the distribution of weights that are placed on things like publications? Are there some factors that tend to be overlooked?

I think that our metrics for success are causing society to miss out on whole categories of successful people.

For example, we spend a lot of time evaluating work and evaluating people but we don’t spend a lot of time evaluating the evaluation processes themselves. There’s no one whose job is to make sure that conference review processes are fair and accurate. We know from the NIPS experiment that there is a lot of noise in the reviewing process (Eric Price has shown that area chairs disagreed more often than they agreed about how to handle a paper in this process), and yet there is no one spearheading efforts to develop better reviewing processes backed by evidence that they actually work. The research community should value efforts that improve the effectiveness of the community as a whole, but so far we just don’t seem to have any way of putting value on such efforts.

9. Do you think machine learning as a field has a different relationship with failure than other fields? Does this affect different groups of people in different ways?

Machine learning has very high expectations in terms of very rapidly producing a lot of successful work and exerting influence over the firehose of everyone else’s rapidly produced work. For example, Ilya Sutskever has over 50,000 Google Scholar citations, while in mathematics, none of the four most recent Fields Medal winners has over 5,000 citations. It is very strange that in our field success is so explosive. This is probably in part because we use arxiv.org so much rather than being more primarily focused on peer-reviewed publications. To be honest, I don’t know a lot about how it affects different groups of people.

10. What are your thoughts on negative results in machine learning?

I think it’s hard to extract value from negative results in machine learning because it can be so hard to tell what caused the negative result. A negative result might point to something very fundamental wrong with an idea, but it might also just be the result of a very small software bug, the wrong idea of the hyperparameter values to try out, too small of a model, etc.

11. If you could reach all senior academics in ML, what would you tell them?

If I could reach all senior academics / conference organizers / journal editors in ML, I’d tell them: The community needs to have a better way of settling disputes over sharing credit for ideas.

Currently, these are mostly settled by the person who feels they have not been given appropriate credit directly contacting the author of the publication that fails to give them credit. This works if the issue was a simple oversight (author of the new publication wasn’t aware of the old publication) but usually if the two parties disagree it can turn ugly. When there’s no central authority and persuasion fails, individuals fall back to a carrot or a stick, and most people do not have much of a carrot to offer.

This kind of experience is especially stressful if a senior, famous professor demands credit from a junior researcher, such as a PhD student.

As my work has become more well known in the machine learning community, I’ve spent more and more of my time dealing with this kind of conflict.

It would be much better if a conference or journal offered a centralized place to have these conflicts adjudicated efficiently by neutral third parties.

12. What is the best piece of advice you could give to your past self?

I wish I’d used some of those GPUs I bought for deep learning to mine some bitcoin.

Przemysław Pawełczak is a tenured assistant professor in the Embedded Software group at TU Delft. He obtained his PhD in wireless communications (topic: Cognitive Radio) at the same university in 2009. Before joining the Tenure Track of TU Delft, Dr. Pawełczak was a postdoctoral scholar at UCLA (2009-2011) and a research scientist at Fraunhofer HHI Institute in Berlin (2012). He is the recipient of 2013 NWO Veni grant. You can find him on LinkedIn, GoogleScholar and Twitter.

1. Thanks for joining How I Fail! Can you tell us a little bit more about yourself?

Great to be invited to this series! I’m an assistant professor in the Embedded Software group at TU Delft since 2013. I enjoy my work a lot. The ability to work and interact with diverse, extremely bright and open-minded people at my university makes me wake up every day and go to work. I’m Polish (and very proud of it) but spent almost all of my professional part of my life abroad (Netherlands, USA, Germany), and only at research institutions. This allows me to have a broad (not Dutch-centric) perspective on how things are handled in the academia.

2. Are there any “notable failures” you would like to share? “Most rejections”, or perhaps a failure that hurt the most?

There is no specific rejection that hurt me the most. After you got one rejection and another (and another) you start to get use to it and accept is a natural part of your trade. Of course, every rejection makes you feel bad. It is very hard to dis-associate criticism about your paper/grant application from attack ad hominem, so reading emails starting with the sentence “we regret to inform you” do feels bitter. What I realized, though, is that as you get older (or as you are longer in the business) that bitter feeling is becoming lesser and lesser. When I was a first year PhD student I could not sleep for days after my first-ever paper rejection, but now I embrace it and use it as an opportunity to improve my work and make it better (if I still can).

3. Do you keep track of your failures? Why/why not?

Yes, I do keep track of my failures. For instance, my “H2020 Rejected Proposals” folder has (as of now) eight sub-folders inside. Why do I keep track of my failures? Because it is good to see how much effort you put into papers and proposal preparations. Also tracking this lets you improve your time allocation and strategic decision-making (for instance: which proposals are worth the effort to re-submit and which are not).

4. Do you think this issue is more prevalent in academia than in other fields? What about differences between academic fields or different countries?

My core observation is that academia (anywhere in the world) is no different from other fields. Everyone competes for scarce resources (in academic terms: limited spots in conference proceedings and journal issues) and there are simply to many of us academics for available top spots. So naturally you will be rejected. Embrace it and be persistent. To say it jovially: law of large numbers (resubmit, resubmit, resubmit) will eventually help you.

5. Is failure/rejection in academia something you discuss with students that you supervise?

All the time! I am actually preparing them for rejection from the first day of their work. Statistically speaking there is a much higher chance that any work will be rejected, so it is critical to psychologically prepare students for this (quite common) event. And since I am the person that sees glass half empty (rather then half-full), projection of my defeatism makes it even easier.

6. What do you do when you receive a rejection?

I am a person that over-excites about many things (either failure or rejection), so to keep my head clean I run. I find running more of a meditative ritual, rather than a physique-improving task, I started running during my second year of my PhD and I run ever since. My PR is 48:27 for 10k, which proves I do not run for the records.

7. What about when you receive good news? Any traditions/rituals there?

See my earlier answer. Whenever I can, I go with my wife to a restaurant to celebrate. Also, I thank my students for their efforts (success is never a lonesome adventure) by inviting them to spend some time together outside the university.

8. Are there any opportunities you didn’t take that you wish you had (so you could add them to either your “success” or your “failure” CV)?

I was too late with my NWO Vidi grant proposal writing preparations, so I had just one chance to apply, which ended up in rejection. All the EU proposals I mentioned earlier that were rejected I consider to be a pure lottery. There were naturally some other (less critical) failures but overall I am happy with the choices I made.

9. Can you think of something you did that felt like a success, but you wouldn’t find in a “success” CV?

Success is built on perseverance and daily progress, so the list of mini-successes would be long… but certainly not worth putting in your CV. Would successfully submitting a paper at 5 AM in the morning after a 24 writing streak would be something worth putting in your CV? I’m doubtful.

10. If you could add a compulsory section to all CVs you receive and send, what would this section be called?

If we could measure “grit” (how persevere one is in reaching ambitious goals despite all setbacks) effectively and objectively I would be happy to see it in all CVs.

11. What is the best piece of advice you could give to your past self?

Accept that nobody (including yourself and all academics) is perfect, so embrace that things around you will not as fast and professional as you would anticipate.

…

For this How I Fail post I have the pleasure of interviewing Emanuele (Manuel) Trucco, MSc, PhD, FRSA, FIAPR. He is the NRP Chair of Computational Vision in Computing, School of Science and Engineering, at the University of Dundee, and an Honorary Clinical Researcher of NHS Tayside. He has been active since 1984 in computer vision and since 2002 in medical image analysis, resulting in more than 250 refereed papers and 2 textbooks (one of which an international standard in his days, with >3,000, Google Scholar Jan 2017).He directs VAMPIRE (Vessel Assessment and Measurement Platform for Images of the Retina), an international research initiative led by the Universities of Dundee and Edinburgh, and several large EPSRC and NIHR projects on retinal biomarkers and diabetes precision medicine. Recent projects have focused on robotic hydrocolonoscopy and whole-body MR angiographic data. Manuel’s hobbies (too many to do any well enough) include playing music (guitar, Scottish bagpipes), reading (avidly), drawing cartoons (not often enough), cooking (often) and running half-marathons (not every week, thank you).

1. Hi Manuel, thanks for joining the How I Fail series! Could you please tell us a little bit about yourself?

I was born in Genova, Italy, birthplace of Cristoforo Colombo who discovered America and is therefore indirectly responsible for consequences like Donald Trump. I live in Edinburgh, deemed the “Athens of the North” during the XVIII-century Scottish Enlightenment, and still a wonderful place to be for academics (three universities, various colleges), lawyers, tourist operators and, well, tourists. I work in Dundee, home of two universities and of the Tay Bridge immortalized by the unbelievably terrible poem by William MacGonagall “The Tay Bridge Disaster” (if you think you cannot write poetry, take a look). I have three children, now old enough to accept my many idiosyncrasies without being openly embarrassed; or perhaps they have learnt to disguise embarrassment effectively. I like people more than roles, and the well-known quote “it is incredible how much you can achieve when nobody claims the credit”. I try to organize my research group, and any group I am involved with, accordingly.

2. Could you share some of your own memorable failures with us?

I imagine you are asking about failures in my work career. The first thing which springs to mind is not grasping occasions immediately when they appear. I recall the very good reviews we got when my and Alessandro Verri’s book appeared in 1998 (yes, I am that old). That created a number of opportunities that we never embraced, including a second edition; we could have done several, thinking about it. Another thing, and a very important one, is recognizing the border between being realistic and undervaluing yourself. I think many of us can do more than we think ourselves capable of. Par contre, there seem to be people who think they have done much more than they have actually. We are all different, and clusters of people and collaborations form accordingly.

3. Do you in general keep track of your failures somewhere? If yes – how and if yet, why not?

I do not keep a written track or anything of the sort, but try to learn from mistakes. I think significant failures stay with us anyway, and we must learn to cope with them. I cannot help thinking of “significant” as meaning “having an influence on our feelings”. That to me does not include a paper rejected by an important conference, normally, but (for instance) being disappointed by a friend – or by myself – in something serious.

4. What do you think about the “CV of Failures”?

I think two things are important. One is to learn from failures so that we avoid similar mistakes in the future. The second is not to obsess about failures, which may lead to depression and underestimating ourselves. That is a very serious risk. We all fail every now and then. We must always look at the bright side of life (yes, this is a citation).

5. Do you have some process/ritual of dealing with failure? Has this process changed throughout your career?

Of course: I wait for the next full moon, I climb the nearest hill and dance waiving a ritual wooden stick. Not really. I cannot say I have a process or ritual. I have realized that the way we deal with failures depend on many parameters, including the kind of failure, your age, your experience, the people who support you, and so on. I can see that time plays an important role: you do change, and there are ways of controlling this change, but not all of it. A good thing is that you get a higher-level view of the world, and you should be able to contextualize better whatever happens. A bad thing is that you have to stop eating a lot of cheese.

6. What about when you receive good news?

I am a natural collaborator, so good news at work often involve other people, directly or indirectly. I love to share good news and normally organize little or big celebrations.

7. Do you think there are differences between how failure is perceived between different countries, different fields, or perhaps across different career stages?

Goodness! Definitely, all of it. Age makes a difference as the kind of things which may fail and the way you face failure both change. The farther you go in your career, the larger the initiatives you get involved with, be that money or impact or intellectual magnitude of your work. So failure may be larger and more impactful later in your career, and affect more people: if you have a team of contract-paid people working with you, procuring money to keep them employed is your responsibility, for instance. It is difficult to generalize, but it looks like work ethos does change across countries and institutions within a single country. I think a significant amount depends on personal temperament: some of us are truly driven by their work (and normally ipso facto excel at it: recall the 10% inspiration and 90% perspiration adage), some prefer a balance between work and other parts of life; some are born leaders, some born followers (which does not means they are less important: no leader would achieve anything without capable followers). And so on.

8. Would finding somebody’s CV of failures online affect how you assess them, for example if you are a reviewer for a grant?

This is a complex and indeed delicate matter. I think I would not personally look actively for a CV of failures, and suspect that, in the UK at least, one would have to be very careful with what information is considered to employ people. Then there is unbalance: it takes some self-confidence to air one’s failures in public, so some people would do it, some would not, creating probably a bias. So I would not look for a CV of failures unless it were made mandatory for everybody (alas, unlikely).

As to grants, I believe they should be (and are, I hope) assessed both for the value of the work proposed and for the track record of the proposers. In this sense failures do affect decisions, although a track record includes what one has done more than where one has failed. You can always think that a proposer should have done more than they declare, of course, but this is different.

9. Are there any opportunities you didn’t take that you wish you had (even if you would probably add them to your hypothetical failure CV)?

Yes indeed. I feel I have missed serious opportunities in my life, both at work and elsewhere. I could never forget a line in Italo Calvino’s Mister Palomar: “his life looked to him like an uninterrupted series of missed occasions” (my translation from Italian). My life does not look to me like that, fortunately, but there were several big occasions missed. I think it is hugely important to have good advisors, especially in the early stages of your life. “Early stages” depend on what you do, so they extend much longer than we imagine. And open, good-humoured friends are invaluable – always.

10. Can you think of something you did that felt like a success, but you wouldn’t find in a “success” CV?

All successes you achieve in your non-work life qualify. Seeing my children starting to achieve their own success in their lives, independent of me, is one of the best experiences I can recall. I hope there will be much more if it. I also believe that maintaining a strong sense of humour in life is a huge asset. If I have one, of course, is not for me to say (unless I develop a split personality).

11. What is the best piece of advice you could give to your past self?

Stop! Think. Take time. Is this really the best course of action? Have you considered the consequences for you, and for others?