CV of Failure: introduction

by
Image by https://unsplash.com/@tersh4u

My CV of Failure

Here it is – my CV of failure, or “shadow CV”.

I first found out about the concept of a CV of failure from this article. After a professor from Princeton posted his CV of failures online, shadow CVs have been getting more attention on Twitter, under the hashtags #ShadowCV and #CVofFailures. And it’s getting very popular too – the same professor now added a “meta-failure” of his shadow CV getting more attention than his research.

I already wrote about various successes and disappointments during my PhD (during my 3rd and 4th years). To write those posts, I used an Excel sheet that I normally use for yearly evaluations. Here is a screenshot from my 3rd year as a PhD student:

excel_progress

The one thing you can probably guess is that green is something that was successful, and red is something that was not. Creating a shadow CV would be essentially compiling all the red parts, over the five years that I’ve been doing research. In the era of tracking everything from what you ate to what music you listened to, why not track failures as well?

The experience

Given that I already had all the data, compiling the CV was quite easy. It was exciting – I was curious whether my shadow CV would be longer than a professor’s. It was comforting – the list wasn’t too long after all, and the inevitability of the list expanding in the near future didn’t seem as daunting. I also realized that it was good to start failing early with travel scholarships, because I feel more prepared now for the larger failures that I encounter.

But most importantly, compiling the CV was motivating. I thought about whether anything would have been different for me if I had seen such CVs a couple of years ago. As many other PhD students, I was not very confident. There were many things I didn’t even dare to apply for. Sometimes senior researchers would tell me these thoughts are unfounded, that I should just apply, and that everybody gets rejected. Sometimes I listened, and sometimes got rejected, but sometimes got accepted, which ultimately gave me more confidence. I hope that seeing shadow CVs can help other students do the same: go for more opportunities, fail, and learn from it.

My relationship with blogging, part 2

by

As I wrote a few months ago, I have a difficult relationship with blogging. In short: I start, after a few months I think what I wrote is silly, and then I get rid of it. This time I promised myself and the internet that I wouldn’t do this. I made no promises about posting great content or posting often, just that I would accept my posts they way they are.

I dare to say that it’s actually going quite OK! I think the thing that is different this time is being on Twitter:

  • I read more blog posts in general, which helps me improve my writing and gives me ideas on what to write about.
  • I realized there are a lot of people struggling with writing, and that ways to improve your writing (such as doing it, as I am doing now!) are a good thing.
  • I connect with more researchers, and am slowly starting to realize my posts might be useful to others

As part of “blog relationship therapy”, I decided to also be more accepting of posts from my earlier blog – the posts I decided were silly in one way or another, and eventually led to that blog’s doom. I’ve resurrected a couple of them. They are mostly about “first” experiences as a PhD student, such as preparing for a lecture or writing a proposal. Enjoy!

On choosing to do a PhD

by

This week I gave a talk at the career event of my former student society for mathematics and computer science students, ‘Christiaan Huygens’ (CH). All the speakers were asked to talk about their work, and the choices they made to get there. As it wasn’t always my goal to do a PhD, I thought it would be good for the students to hear about the doubts that I had. And now, for the purpose of sharing N=1 experiences, I’d like to share these thoughts with a wider audience.

Choices

One of the courses I followed during my masters was Pattern Recognition (IN4085, for the readers from TU Delft). I was immediately sold. It seemed magical that reading licence plates, recognizing faces, or predicting a patient’s diagnosis, were all based on the same underlying principles. I followed all the courses I could find on related topics, and did my graduation project on a pattern recognition topic (here is proof).

I was convinced that my job had to do something with pattern recognition. But, I also wanted to somehow apply all the other skills that I had developed during my student union time. As many of my classmates were choosing consultancy jobs, I was excited to find out that there was something similar there to match both my interests: a data analysis consultancy job at a large company in Amsterdam. Note that this was 2010, and nobody was hiring “data scientists” yet, at least in the Netherlands.

Everything was going really well with that company. I had visited them to find out more about their projects and meet the team. I really loved the new environment, and I was convinced I would be hired after an interview. Shortly afterwards, an opportunity to do a PhD in Delft came up. That was also an environment that I loved, but it was entirely different from the job in Amsterdam. It was safe and familiar. After all, I thought that during my MSc project I had already figured out what research was about (cough cough). Fueled by the belief that you should always do new and challenging things, I felt that a PhD would be seen as an inferior choice.

Doubts

But, doubts also started to creep in. I could not define them fully at the time, but now I am quite sure the doubts were: (i) I wasn’t sure I would be challenged enough by the technical challenges of the job in Amsterdam and (ii) I thought I would be challenged too much socially — always looking professional, always interacting with people. All things that I enjoy, but perhaps not every day, and not coupled with a long commute.

Perhaps my biggest problem was that at the company, it would be part of my job to oversell things — make things seem more impressive than they are to clients, and come up with convincing arguments on the spot. This ability was considered very valuable among my classmates, and I imagine quite sought after by companies. I of course felt honored by the company’s belief I had this ability, because it wasn’t something that came to me naturally. So while I was excited and challenged by the prospect of developing this ability, it also felt like I would be betraying myself a little bit.

Best advice I received

One thing that helped me in my decision is a conversation I had with a former student. He had just finished his PhD and was switching to an industry job. The advice was to

  • compare concrete opportunities (rather than PhD vs not PhD)
  • consider the people I would work with and the daily tasks I would have to do
  • whether I could be myself, and how the position would fit in with the rest of my life

This cleared things up for me, and once I had accepted the PhD position, it felt like a huge weight off my shoulders.

I didn’t regret my decision for a second afterwards. I had a great time during my PhD because of the people that I worked with, not only inside the lab, but also other researchers that I met at conferences. I discovered I was very wrong about knowing what research is about! There are new, exciting and challenging things in my job every day, but challenging in a way that is meaningful to me.

I also had quite a lot of freedom, both in the ideas I pursued and in managing my time. I was in the office during regular work hours, but I appreciated being able to take a day off, go on holiday, or work at home if I wasn’t having the best day. I felt that I was being valued for my ideas, rather than for showing up. And isn’t being valued one of the most important components for job satisfaction?

Keep in mind

That is not to say that you should always choose a PhD when in a similar situation. There are huge differences in PhD positions as well! Not all researchers are nice people, and not all projects offer the same freedom that I had. I hope that this strategy — considering concrete opportunities, and staying true to yourself — should help you with the answer.

Why you should post preprints on arXiV

by

Recently on Twitter I saw a lot of discussions about preprints, such as under the #ASAPBio hashtag, which originated in the biology community. My guess is that preprints are more or less common in different fields, and I thought it was normal for Computer Science to do it, so I couldn’t contribute anything to the topic. But I’ve encountered some doubts when I encouraged other CS students to upload their work to arXiV, so I thought I’d share my N=1 experience with preprints.

Long story short, I spent a good part of 2013 writing journal papers. I submitted three of them that year, and directly uploaded the submitted versions on arXiV. You can see my page on arXiV here.

I spent 2014 revising these papers. One paper was accepted in 2014, and two others only in 2015, when I was already a postdoc. One of the accepted papers is still in press, even though it is already 2016. I imagine it will be three years (!) between my initial submission — that really isn’t that different from the revised version — and the published version. And this is in Computer Science, a fast-moving field!

As a PhD student / postdoc / aspiring researcher, you can’t really afford such a time lag. And that is where preprints have been immensely helpful to me in different ways:

  • Two of the papers were based on earlier conference papers. When I was discussing that work with other researchers (at conferences, via email), I could send them the preprint, which contained more detailed results.
  • The third paper (a type of survey) was completely new, and I was a bit scared that somebody would publish something similar before me. The preprint was actually a way to assure myself that it was now documented that I came up with the idea. Again, I discussed this work with other researchers while it was already in arXiV, and even got some valuable comments, which helped me a lot when revising the paper.
  • The preprints were cited (mostly by myself, but also by other researchers). After publication, I merged each preprint with the published version in Google Scholar. I don’t really have a lot of citations, but I would have had even less if the papers only became available in 2015 instead of 2013.
  • I didn’t apply for jobs while I had any unpublished preprints, but if this was the case, I could put the preprints on my CV, which is more informative than simply listing the paper title with the comment “manuscripts in preparation”.
  • Most journals allow this! You can check on this website what your journal’s policy is

If you are a student in Computer Science (or anything, really) and you are doubting about uploading a preprint of your recent work, I hope this might change your opinion a little bit.

My relationship with “save for later”

by

Just like blogging, using “save for later” is another thing I have trouble with. I come across a lot of awesome things online, from articles to interesting blogs to cat furniture ideas. Despite having access to several tools to organize such gems (from “Like” on Twitter, to Evernote to Pinterest), I am not really happy with my current setup. I do save things “to read later” in various ways, but the “later” part almost never happens.

Perhaps the only exception to this rule is how I deal with research papers relevant to my projects. When I come across a relevant paper, usually through a Google scholar alert, I immediately include it in the ShareLateX project on that topic. Perhaps that part by itself requires some explanation: I start a ShareLateX project very early on for each topic I am working on, and eventually that document grows into a paper. Here is a screenshot of my most recent projects:
sharelatex_projects

For me this is a foolproof way to remember these relevant papers. I do not forget my projects, and when I pick one of them, either to brainstorm what to do next or to write parts up, I WILL scan, then possibly print and read those papers.

I’ve thought about the differences between this system, and what I do with all the other articles, blogs, etc that I save for later (and that I’m too embarrassed to make a screenshot of). There are really only two that I could think of:

  • The place. For other types of content (anything that is not an article not related to my research) I use the bookmarks folder, Evernote (if related to research in general, academia, personal development), Pinterest (if related to food, exercise, travel). As you can guess, none of these places are places I review every so often.
  • The purpose. The research articles have a clear purpose: “read, summarize and reference in this paper”. Most of the other content I save could probably be labeled as “might be interesting once I get around to it”, which is not really a purpose. The current way I try to organize all those items is by topic, such as “machine learning” or “productivity”. Each topic will include items I’ve already reviewed and saved for some reason, or those I still want to read. Perhaps categories such as “read if bored on the train”, “use as reference in grant proposal”, “write about in blog post” would be more effective.

And those categories are actually something I will try to implement this year! The last one in particular should be interesting: I really dread organizing my favorites, and I find it difficult to decide on blog post topics — so, why not try an approach that has already worked for me elsewhere and kill two birds with one stone? I just need to decide on the place – ShareLateX does not seem really appropriate this time. Don’t forget to check in later to see the results!

Stable detection of abnormalities in medical scans

by

This is my second post about a paper I wrote. This time it is about Label stability in multiple instance learning, published at MICCAI 2015. Here you can see a short spotlight presentation from the conference. The paper focuses is on a particular type of algorithms, which are able to detect abnormalities in medical scans, and a potential problem with such algorithms.

What if I told you that I like words “cat”, “bad” and “that”, but I don’t like the words “dog”, “good” and “this”. Did you spot a pattern? To give you a hint, there is one letter that I like in particular*, and I don’t like any words that don’t have that letter. Note that now you are able to do two things: to tell whether I’d like any other word from the dictionary, AND why that is.

You are probably asking yourself what this has to do with medical scans. Multiple instance learning (MIL) algorithms out there that are faced with a similar puzzle: each of the scans in group 1 (words that I like) has abnormalities (a particular letter), and each of the scans in group 2 is healthy. How can we tell whether a scan has abnormalities or not, and where those abnormalities are? If the algorithm figures this out, it could detect abnormalities in images it hasn’t seen before. This is a toy example of how the output could look like:

lungs1

Detecting whether there are any abnormalities is slightly easier than finding their locations. For example, imagine that the scan above has the lung disease COPD, and therefore contains abnormalities. Now imagine, that our algorithm output has changed:

lungs2

The algorithm still correctly detects that the scan contains abnormalities, but the locations are different. If the locations of the abnormalities were clinically relevant, this would be a problem!

Of course, in the ideal case we would evaluate such algorithms on scans, where the regions with abnormalities are manually annotated by experts. But the problem is that we don’t always have such annotations – otherwise we probably would need not to use MIL algorithms. Therefore, the algorithms are often evaluated on whether they have detected ANY abnormalities or not.

In the paper I examined whether we can say something more about the algorithm’s output, without having the ground truth. For example, we would expect a good algorithm to be stable: for one image, slightly different versions of the algorithm should detect the same abnormalities. My experiments showed that an algorithm that is good at detecting whether there are any abnormalities, isn’t necessarily stable. Here is an example:

copd stability

Here I compare different algorithms – represented by different colored points – on a task of detecting COPD in chest CT scans. The y-axis measures how good an algorithm is at detecting COPD (whether there are any abnormalities) – the higher this value, the better. Typically this would be the measure of how researchers in my field would choose the “best” algorithm.

I proposed to also examine the quantity on the x-axis, which measures the stability of the detections: a value of 0.5 means that multiple slightly different versions of the same algorithm only agree on 50% of the abnormalities they detected. Now we can see that the algorithm with the highest performance (green square) isn’t the most stable one. If the locations of the abnormalities are clinically relevant, it might be a good idea to sacrifice a little bit of the performance by choosing a more stable algorithm (blue circle).

A more general conclusion is: think carefully whether your algorithm evaluation measure really reflects what you want the algorithm to do.

Year in review: final year as a PhD student

by

This post is a summary of 2014, the last year of my PhD. I am writing it a whole year later due to my difficult relationship with blogging. There are two reasons for this: a recent conversation about blogging on Twitter, with this result, and the fact that the summary of my third PhD year played an important role in me deciding to resurrect this blog.

As 2013 was a year of submitting papers, I expected that 2014 would be a year of paper resubmissions. That guess was quite accurate. But 2014 had more challenges in store for me. The year didn’t start out great for me for personal reasons. I am not sure I will ever discuss the details online, so let’s just leave it at “life changing event”. Up until that point, I was sure I would finish my PhD on time. But, with so many things changing so rapidly, I started having serious doubts about my progress.

Writing and staying motivated

Despite the personal chaos, I continued to work on the revisions of my rejected papers. In February, I resubmitted Paper 1. That was tough, so I didn’t want to touch the other rejected papers for a while. Besides, I had other activities lined up, such as a research visit to Copenhagen, where I wrote a conference paper about the work I had done the year before. The visit was a great experience, both professionally and personally! Unfortunately, I received a rejection, adding yet another thing to the revise-resubmit list. On top of that, I was rejected for the Anita Borg scholarship for the third and final time. But there was also a bright side: for example, around the same time I gave my first invited talks, which was a much-needed boost for my confidence.

In June, I finally received the coveted “We would be happy to publish your manuscript” email about Paper 1. This gave me the needed motivation to continue with the other revisions. In July, I resubmitted Paper 2, and in September, Paper 3, which by then had already been rejected at two different journals. Again, it was very helpful to be involved in other activities, such as organizing a workshop and teaching, to stay motivated.

With one accepted journal paper and two others under review, I again started hoping that I would submit my thesis by the end of the year. The thesis requires at least four chapters, each based on a “publishable” paper. My supervisors agreed, so I spent the last months working on Paper 4. Paper 4 described recent results, and was therefore very refreshing in the midst of all the revising. I finished it on time and submitted it to a conference in December. And then, with three papers “in limbo”, both 2014 and my PhD contract, ended.

Take-aways

My year of revisions had a few successes and several disappointments. However, the more important successes were the things that these experiences taught me. I…

  • …became a seasoned reviser-and-resubmitter
  • …learnt how to stay confident as a researcher despite a lot of disappointments
  • …realized even more deeply how important it is to have colleagues who believe in you, who support you, and who are up for a grabbing a beer (or a Spa rood), whether it is to celebrate or offer a shoulder to cry on.

My relationship with blogging

by

I have a love-hate relationship with blogging. I have always enjoyed having some sort of website. When I was 10 or so, my dad showed me how to build websites in HTML, and I made a website about the Spice Girls. There was no original content on the website, but the fact that I had a website and could update it if I wanted, that’s what counted for me. Of course, when I got a bit older and became embarrassed by my choice of music, the website stopped existing. In high-school, I got a bit more interested in webdesign. Blogs were becoming popular, and since I didn’t have any particular hobbies, I made a website with a blog about what was happening in my life. You can already probably guess what happened… I got older, decided my problems from a year or two before were very silly, and that website disappeared as well. Which is too bad, because now I would find it interesting to see how I thought about things in 2004.

In university I had a break from websites and blogging, probably because my desire to “do something with websites” was satisfied by my part-time job. But when I started working on my MSc thesis, something started nagging at me again. I was learning more about doing experiments, reading and writing papers, and wanted to share my thoughts. Perhaps that was the first time I felt that I had content worth sharing, so I started a blog again. In the end, I often felt obliged to post “something”, which resulted in rather uninteresting posts. This also happened during my PhD – I got inspired by website such as PhDTalk, but my attempts were never really quite successful, because I didn’t spend enough time on them. Again, my earlier posts just seemed silly to me, especially after some major changes in my life. My website was offline once again.

As a postdoc, I’ve started reading more and more academic blogs, and since a few months, I even have a Twitter account. So again, I want to have a blog, and I regret not doing a good job with the other ones. But a difference between regretting getting rid of your website in 2004, and getting rid of your website in 2014, is that I’ve been with the same host for the past 5 years or so, and could recover any content that I posted. So, I have decided to resurrect my blog a little bit, including posts from earlier editions. I’m not making any promises about how often I will post this time, but I will try to keep myself from going through the whole delete-regret cycle. Stay tuned!

Year in review: third year as a PhD student

by

As I mentioned before, it’s important to keep track of your successes and disapointments. Since I do have a list of sorts, I decided to share my summary of 2013 here.

Writing

2013 was definitely a year of journal papers. Or at least, of long overdue journal paper submissions. Here are the totals! I submitted four times in total (one paper twice, and two papers once). Two of these were rejections, one “revise and resubmit” and one still under review. So, 2014 probably will be a year of journal paper resubmissions.

Reviewing

Next to paper writing, there was also paper reviewing. In the beginning of last year, I was getting worried that I was not invited for reviews, but this worry turned out to be unfounded. I guess this goes together with submitting journal papers (and getting into the system) and meeting more people, who have more reviews than you, but are also more busy. I want to believe in review-karma: by writing good reviews, I hope to get good reviews. By good, I mean objective and constructive, not necessarily an “accept”.

Funding

2013 was also a year in which I tried to apply for scholarships to finance my conference visits and the trip to Tuebingen. For the second time (the first time being in 2012), I did not get the Anita Borg scholarship. I did get the ACM-W / Microsoft Research grant to go to a conference in China, which was awesome! The application that I spent quite a lot of time on, for the short-term fellowship from EMBO to go to Tuebingen, unfortunately got rejected (after I returned from Tuebingen already). However, I was able to get some financial support through my university, which was not a competitive application, but very helpful.

Research visit

And of course, 2013 was the year I went on a research visit, for which I have not (yet?) been able to write an overview. In short, the three months went by really fast and I had a great time. What everybody says about research visits is true. It is really helpful to experience a different place and get an idea of how people do research there. I think it’s a must for all PhD students, especially from smaller labs. It probably doesn’t even need to be a lab in a different country to get an impression of “how things are done” and to pick up useful research skills. I already have my next short visit planned, what about you? Did you / will you do a research visit during your PhD?

On recording your progress

by

It’s been almost a year since I started to blog alongside my PhD. I’m not sure whether I mentioned this a year ago, but my initial goal was to write something every week, which quickly deteriorated into writing something every month. With this post, I have been able to keep the latter promise up.

Although I had many moments of “oh, this is something I could write a post about!”, only few of them actually made their way from my thoughts into a digital version. The main reasons for this are, I think:

– too little practice: I really do find it difficult to write something that’s not directly about my research

– too little privacy: there are some issues I would not want to discuss online because my name is linked to my blog and I don’t want my opinions to always be “out there” somewhere. Many of the blogs I find very interesting (not just the ones that are linked on this page) are actually blogs that discuss such issues, and for 95%* these are anonymous.
* I initially wrote 95\% which amused me quite a lot.

– too little expertise: I like posts which contain advice on how to do something better, such as never worrying about poster transport again. The truth is, however, that there are not a lot of things I feel I’m more knowledgeable about than other researchers with blogs.

– too little information: I am, of course, knowledgeable about what I do on a daily basis (submitted this, got rejected for that). Of course, this is mostly relevant to me and not to readers in general. I am aware this is my blog and I can post whatever I like, but I am less motivated to spend time writing something that is not helpful to others. Also, I

What I recently realized is that I would have enjoyed to have more of these “progress” posts, just for myself. In my first attempt at blogging, I wrote about how I submitted my first paper and how a few months later, I got the email that started with, “We are pleased to inform you…”. Or about how I reviewed the paper for the first time, and it turned out to be horrible. It’s nice to remember how I felt then, what my goals were, and how I generally thought about research.

My advice (this is actually a helpful post!) if you are a PhD student: write your successes and disappointments down somewhere. Not necessarily in a blog. Maybe it’s even better if it’s not a blog, you might spend less time worrying about how to write it down, and who is going to read it. But sometime later, you will enjoy reading about these experiences, and what they tell you about your progress. Happy writing!

Mastodon More Mastodon
%d bloggers like this: