Defending propositions: timeline and the role of graduate school

This is the second post in the Propositions series. The first post is here.

Timeline

Before I dive into the propositions, I want to give a bit of context to how the propositions evolved over time. I show the evolution in the timeline below. The rows are different proposition ideas and the columns are different versions of propositions.pdf I could find in my Dropbox. Red propositions were ultimately rejected. Orange and yellow propositions “made it” but in a modified form. Green propositions are very close to the final versions that I posted last time.

propositions-evolution

As you can see, some propositions survived the whole journey from the first draft in April 2014, to the approved version in January 2015. Today I want to talk a bit more about the proposition on the role of graduate school, which is one of the ideas that I had on my mind the longest.

The proposition

“Graduate school should be a resource, not a requirement”

When I went to university, a lot of things were new, but a few things were exactly how I imagined. You could decide whether to go to class, whether to do an assignment, whether to show up to the exam. It was all my own responsibility, and I loved it! When I got a job – my PhD project, as PhDs are employees of the university – I assumed the same responsibility for my project and my personal development. It was up to me to make sure that four years would lead to a thesis as well as transferable skills for my next job.

But the situation was not the same for all students. In my second year of PhD, the university introduced the graduate school – a program that “ensures and enhances the development of scientific quality along with the needed proficiency for interpersonal skills“. All PhD students could attend the courses, but new PhD students had stricter requirements (compulsory course X, at least 1 course from module Y).

I was happy I could choose from many courses, and signed up for several. Unfortunately my experiences with the compulsory aspect were not as positive. For example, I very excitedly joined a teaching course. In the lessons we would split up into small groups and discuss our experiences with teaching. I shared my experiences with my group, only to hear “sorry, we are only here for the credits” back. You could also earn credits with “on the job” activities like joining a reading group, giving a presentation or collaborating with another student. Taking this to the extreme, you could start a PhD with a list of things to do, and simply tick all the boxes.

Despite my disinterested fellow students, I did learn something in the teaching course: to align learning objectives and activities. But if the learning objective is to become an independent researcher, is ticking boxes the best activity to do?

Defending propositions : introduction

Propositions?

In the Netherlands, along with your thesis you defend a few – let’s say 10 – propositions. Propositions are statements that are “opposable and defendable” and cover a number of topics. The first few are usually about the topic of your thesis, but the others can cover pretty much any topic.
These last few propositions are usually the most interesting, as they resonate with everybody – not just people who are acquainted with your research topic. It is a way to show your personality, by voicing your concerns about a particular topic, or even by slipping in a bit of humor. But apart from being a creative outlet, propositions are also rumored to be difficult to write. In this series of posts I share my experiences with writing propositions, which might give you some inspiration for writing yours.

My propositions

To get started, I present to you my propositions:

propositions
[PDF]

As you can see, propositions 1-4 are about my thesis and pattern recognition in general. Propositions 5-10 are about other topics, but most relate to doing research. It is these propositions that were the most difficult to come up with, but most rewarding to refine into their final form. In the upcoming posts, I plan to share more about a few of these propositions. I will also write about the brainstorming process that I went through, and what I think about this tradition now that it’s all behind me.

While I’m getting the other posts in this series ready, please let me know what you think about propositions. Is it a good tradition? Does it add something to the PhD defense or is it a waste of time? If you are are doing a PhD in the Netherlands, are you thinking about what your propositions might be?

CV of Failure: Things I didn’t dare to try

Image by https://unsplash.com/@tersh4u

What counts as failure?

A recent #withAPhD conversation on Twitter prompted me to write a bit more about my CV of failures.

So far, I have been tracking the “quantifiable” failures, such as paper or grant rejections in my CV of failures, or shadow CV. However, there are a lot of other things that contribute to my experience of failure (and learning to deal with it) which are more difficult to quantify – because I have not tried them at all.

Most of these things can be summarized with the words “impostor syndrome.” I was convinced I would fail, or perhaps even worse, I was convinced “they” would laugh at me for even trying. But time and time again, evidence showed that I probably did have a chance. And even if I had failed, “they” would have thought it was good that I tried. It probably would have been better than always regretting not trying in the first place. So, what do I regret that I didn’t try?

Things I didn’t try

Although originally I thought only about purely academic things, I realized this pattern of not trying goes back much further. A few examples:

  • Talent show in high school. I’ve played piano for 7 years, but don’t consider it “talent show” quality, so I don’t sign up. At the talent show, somebody else is playing piano, but with many hiccups along the way.
  • First year of computer science at university. The student organization has sign up lists for different committees. The committee to organize parties seems fun, but I’m afraid it will have too many people, and I won’t be chosen, so I don’t sign up. Later I become friends with several guys who did join the committee, and realize they would have loved to have me.
  • Internships during my BSc and MSc. Several people are going abroad for internships. I’m afraid to get delayed with getting my diplomas. The projects I do in the Netherlands are all great and I get my diplomas on time, but the gap between the international experience I have, and the international experience that I could have, starts widening.
  • I’m writing papers during my PhD. The most competitive conferences such as NIPS seem to be way out of my league. I submit to good, but less competitive conferences and workshops. My first three papers all get accepted and I have a great time at the conferences. Later, I get evaluated on the quality of my publications – the places I’ve published do not really “count”. I read more papers from NIPS, and realize that maybe, I could have published there, too.
  • I’m finishing my PhD, and read all the regulations for graduating. A part of the regulations describe the requirements for cum laude. This involves a recommendation from the PhD advisors and several external reviewers. I ponder about asking my advisors, but decide against it. After all, even a graduate from our department who wrote several highly cited papers, didn’t graduate cum laude. “They” would find it ridiculous that I even brought the subject up. My defense is a success and the committee members are very positive. Later I confess to one of my advisors about my doubts, and he reassures me it would have made sense to at least try.

I learn from these events by recognizing the patterns and doing things differently the next time around. I did go on to participate in many committees, and even lead the student organization. The internship abroad was possible during my PhD. Although I still haven’t tried submitting to NIPS, I am now getting rejected often (and occasionally accepted) at MICCAI. I approach senior academics and ask for recommendations for fellowships and jobs. I am still scared every time, but past experiences tell me that it’s really much better if I try, than if I don’t.

Or, to quote Susan Jeffers*,

You’re not a failure if you don’t make it. You’re a success because you tried.

* Author of Feel the Fear… and Do it Anyway. The title alone is great advice.

Firsts: reviewing a paper

After a conversation on Twitter about when to start reviewing papers, I decided to post about my own experiences with reviewing, when and how and when I started, and what I wish I would have known in advance / done differently.

I was in my third PhD year that I started wondering about being a reviewer. I heard of another PhD student being asked to review, and got worried why that has not happened to me yet. But shortly afterwards, I was invited for two journals, and then more and more invitations (including more important journals where I have not published) followed, so the worry disappeared. In this post I share some experiences about what helped me with reviewing, and what I could have done better.

Getting invited

  • I discussed the issue with more senior academics, who were already reviewing for a while. They gave me some pointers about how they got started, but they also made a mental note of me being a potential reviewer. Then, if they were an editor, a program committee member, or simply unavailable to review themselves, they could mention my name.
  • I started submitting journal papers, which meant that I had to create accounts with publishers. Editors were then able to link my name to different keywords, even if my own papers were not published yet. In fact, most of the invitations I received were from journals that I had never submitted to, but that shared a publisher with a “submitted-to” journal. In retrospect, I could have created the accounts even before I was ready to submit, and indicated that I was available to review.
  • I maintained an online presence with the page at my university and Google Scholar. My guess is that editors who did not know me personally, would scan these pages to get an idea about my research and experience. Having a blank webpage (with only your name and contact details) might send the wrong message in such a case.

Accepting the invitation

Once I got an invitation, I accepted it! The paper was on the topic of my PhD and the invite gave me a feeling of accomplishment (and relief), so I felt ready to do it. It could also happen that you do not feel ready, for example because it is unusual for PhD students to review papers. My advice is to go for it anyway. Somebody else thought you have the expertise you do it, so don’t listen to the impostor inside you, and give it your best shot! Reviewing taught me a lot about my own writing, so I’m happy I didn’t wait until I was “grown up” to start doing it.

Reviewing

Then it was time for the actual review. I spent quite a lot of time on this, because I was doing the review by myself, and I wanted to make sure I didn’t miss anything. I thought the paper was not very strong, but I felt I couldn’t rely on just my opinion to say that. So, I discussed the approach with my supervisors, who agreed with me.
Then I read a few “how to” guides and started writing what I wanted to be a constructive review. I also went over the reviews I had received for my papers, as both good and bad examples of reviews. This was more insightful than the guides, so in retrospect, I should have asked colleagues for more examples to learn from.

An alternative would have been to ask a supervisor or colleague to go through the review process with me from start to finish. At the time, I probably thought this would be asking too much, but now I feel that many academics would be happy to do this.

Keeping track of reviews

I created a folder called “Reviews” with a subfolder with the name of the journal, and stored the submission and my review of it. This seemed sufficient at the time, as most people listed only the names of the journals on their CV. But it might be a good idea to keep track of what happened to the paper afterwards, should you need that information later.

Update 4th of October 2016: I found an excellent place to keep track of reviews, share reviews and even rate reviews written by others: Publons! I am still in the process of adding reviews to my profile, but you can already check it out here

CV of Failure: introduction

Image by https://unsplash.com/@tersh4u

My CV of Failure

Here it is – my CV of failure, or “shadow CV”.

I first found out about the concept of a CV of failure from this article. After a professor from Princeton posted his CV of failures online, shadow CVs have been getting more attention on Twitter, under the hashtags #ShadowCV and #CVofFailures. And it’s getting very popular too – the same professor now added a “meta-failure” of his shadow CV getting more attention than his research.

I already wrote about various successes and disappointments during my PhD (during my 3rd and 4th years). To write those posts, I used an Excel sheet that I normally use for yearly evaluations. Here is a screenshot from my 3rd year as a PhD student:

excel_progress

The one thing you can probably guess is that green is something that was successful, and red is something that was not. Creating a shadow CV would be essentially compiling all the red parts, over the five years that I’ve been doing research. In the era of tracking everything from what you ate to what music you listened to, why not track failures as well?

The experience

Given that I already had all the data, compiling the CV was quite easy. It was exciting – I was curious whether my shadow CV would be longer than a professor’s. It was comforting – the list wasn’t too long after all, and the inevitability of the list expanding in the near future didn’t seem as daunting. I also realized that it was good to start failing early with travel scholarships, because I feel more prepared now for the larger failures that I encounter.

But most importantly, compiling the CV was motivating. I thought about whether anything would have been different for me if I had seen such CVs a couple of years ago. As many other PhD students, I was not very confident. There were many things I didn’t even dare to apply for. Sometimes senior researchers would tell me these thoughts are unfounded, that I should just apply, and that everybody gets rejected. Sometimes I listened, and sometimes got rejected, but sometimes got accepted, which ultimately gave me more confidence. I hope that seeing shadow CVs can help other students do the same: go for more opportunities, fail, and learn from it.

My relationship with blogging, part 2

As I wrote a few months ago, I have a difficult relationship with blogging. In short: I start, after a few months I think what I wrote is silly, and then I get rid of it. This time I promised myself and the internet that I wouldn’t do this. I made no promises about posting great content or posting often, just that I would accept my posts they way they are.

I dare to say that it’s actually going quite OK! I think the thing that is different this time is being on Twitter:

  • I read more blog posts in general, which helps me improve my writing and gives me ideas on what to write about.
  • I realized there are a lot of people struggling with writing, and that ways to improve your writing (such as doing it, as I am doing now!) are a good thing.
  • I connect with more researchers, and am slowly starting to realize my posts might be useful to others

As part of “blog relationship therapy”, I decided to also be more accepting of posts from my earlier blog – the posts I decided were silly in one way or another, and eventually led to that blog’s doom. I’ve resurrected a couple of them. They are mostly about “first” experiences as a PhD student, such as preparing for a lecture or writing a proposal. Enjoy!

On choosing to do a PhD

This week I gave a talk at the career event of my former student society for mathematics and computer science students, ‘Christiaan Huygens’ (CH). All the speakers were asked to talk about their work, and the choices they made to get there. As it wasn’t always my goal to do a PhD, I thought it would be good for the students to hear about the doubts that I had. And now, for the purpose of sharing N=1 experiences, I’d like to share these thoughts with a wider audience.

Choices

One of the courses I followed during my masters was Pattern Recognition (IN4085, for the readers from TU Delft). I was immediately sold. It seemed magical that reading licence plates, recognizing faces, or predicting a patient’s diagnosis, were all based on the same underlying principles. I followed all the courses I could find on related topics, and did my graduation project on a pattern recognition topic (here is proof).

I was convinced that my job had to do something with pattern recognition. But, I also wanted to somehow apply all the other skills that I had developed during my student union time. As many of my classmates were choosing consultancy jobs, I was excited to find out that there was something similar there to match both my interests: a data analysis consultancy job at a large company in Amsterdam. Note that this was 2010, and nobody was hiring “data scientists” yet, at least in the Netherlands.

Everything was going really well with that company. I had visited them to find out more about their projects and meet the team. I really loved the new environment, and I was convinced I would be hired after an interview. Shortly afterwards, an opportunity to do a PhD in Delft came up. That was also an environment that I loved, but it was entirely different from the job in Amsterdam. It was safe and familiar. After all, I thought that during my MSc project I had already figured out what research was about (cough cough). Fueled by the belief that you should always do new and challenging things, I felt that a PhD would be seen as an inferior choice.

Doubts

But, doubts also started to creep in. I could not define them fully at the time, but now I am quite sure the doubts were: (i) I wasn’t sure I would be challenged enough by the technical challenges of the job in Amsterdam and (ii) I thought I would be challenged too much socially — always looking professional, always interacting with people. All things that I enjoy, but perhaps not every day, and not coupled with a long commute.

Perhaps my biggest problem was that at the company, it would be part of my job to oversell things — make things seem more impressive than they are to clients, and come up with convincing arguments on the spot. This ability was considered very valuable among my classmates, and I imagine quite sought after by companies. I of course felt honored by the company’s belief I had this ability, because it wasn’t something that came to me naturally. So while I was excited and challenged by the prospect of developing this ability, it also felt like I would be betraying myself a little bit.

Best advice I received

One thing that helped me in my decision is a conversation I had with a former student. He had just finished his PhD and was switching to an industry job. The advice was to

  • compare concrete opportunities (rather than PhD vs not PhD)
  • consider the people I would work with and the daily tasks I would have to do
  • whether I could be myself, and how the position would fit in with the rest of my life

This cleared things up for me, and once I had accepted the PhD position, it felt like a huge weight off my shoulders.

I didn’t regret my decision for a second afterwards. I had a great time during my PhD because of the people that I worked with, not only inside the lab, but also other researchers that I met at conferences. I discovered I was very wrong about knowing what research is about! There are new, exciting and challenging things in my job every day, but challenging in a way that is meaningful to me.

I also had quite a lot of freedom, both in the ideas I pursued and in managing my time. I was in the office during regular work hours, but I appreciated being able to take a day off, go on holiday, or work at home if I wasn’t having the best day. I felt that I was being valued for my ideas, rather than for showing up. And isn’t being valued one of the most important components for job satisfaction?

Keep in mind

That is not to say that you should always choose a PhD when in a similar situation. There are huge differences in PhD positions as well! Not all researchers are nice people, and not all projects offer the same freedom that I had. I hope that this strategy — considering concrete opportunities, and staying true to yourself — should help you with the answer.

Why you should post preprints on arXiV

Recently on Twitter I saw a lot of discussions about preprints, such as under the #ASAPBio hashtag, which originated in the biology community. My guess is that preprints are more or less common in different fields, and I thought it was normal for Computer Science to do it, so I couldn’t contribute anything to the topic. But I’ve encountered some doubts when I encouraged other CS students to upload their work to arXiV, so I thought I’d share my N=1 experience with preprints.

Long story short, I spent a good part of 2013 writing journal papers. I submitted three of them that year, and directly uploaded the submitted versions on arXiV. You can see my page on arXiV here.

I spent 2014 revising these papers. One paper was accepted in 2014, and two others only in 2015, when I was already a postdoc. One of the accepted papers is still in press, even though it is already 2016. I imagine it will be three years (!) between my initial submission — that really isn’t that different from the revised version — and the published version. And this is in Computer Science, a fast-moving field!

As a PhD student / postdoc / aspiring researcher, you can’t really afford such a time lag. And that is where preprints have been immensely helpful to me in different ways:

  • Two of the papers were based on earlier conference papers. When I was discussing that work with other researchers (at conferences, via email), I could send them the preprint, which contained more detailed results.
  • The third paper (a type of survey) was completely new, and I was a bit scared that somebody would publish something similar before me. The preprint was actually a way to assure myself that it was now documented that I came up with the idea. Again, I discussed this work with other researchers while it was already in arXiV, and even got some valuable comments, which helped me a lot when revising the paper.
  • The preprints were cited (mostly by myself, but also by other researchers). After publication, I merged each preprint with the published version in Google Scholar. I don’t really have a lot of citations, but I would have had even less if the papers only became available in 2015 instead of 2013.
  • I didn’t apply for jobs while I had any unpublished preprints, but if this was the case, I could put the preprints on my CV, which is more informative than simply listing the paper title with the comment “manuscripts in preparation”.
  • Most journals allow this! You can check on this website what your journal’s policy is

If you are a student in Computer Science (or anything, really) and you are doubting about uploading a preprint of your recent work, I hope this might change your opinion a little bit.

My relationship with “save for later”

Just like blogging, using “save for later” is another thing I have trouble with. I come across a lot of awesome things online, from articles to interesting blogs to cat furniture ideas. Despite having access to several tools to organize such gems (from “Like” on Twitter, to Evernote to Pinterest), I am not really happy with my current setup. I do save things “to read later” in various ways, but the “later” part almost never happens.

Perhaps the only exception to this rule is how I deal with research papers relevant to my projects. When I come across a relevant paper, usually through a Google scholar alert, I immediately include it in the ShareLateX project on that topic. Perhaps that part by itself requires some explanation: I start a ShareLateX project very early on for each topic I am working on, and eventually that document grows into a paper. Here is a screenshot of my most recent projects:
sharelatex_projects

For me this is a foolproof way to remember these relevant papers. I do not forget my projects, and when I pick one of them, either to brainstorm what to do next or to write parts up, I WILL scan, then possibly print and read those papers.

I’ve thought about the differences between this system, and what I do with all the other articles, blogs, etc that I save for later (and that I’m too embarrassed to make a screenshot of). There are really only two that I could think of:

  • The place. For other types of content (anything that is not an article not related to my research) I use the bookmarks folder, Evernote (if related to research in general, academia, personal development), Pinterest (if related to food, exercise, travel). As you can guess, none of these places are places I review every so often.
  • The purpose. The research articles have a clear purpose: “read, summarize and reference in this paper”. Most of the other content I save could probably be labeled as “might be interesting once I get around to it”, which is not really a purpose. The current way I try to organize all those items is by topic, such as “machine learning” or “productivity”. Each topic will include items I’ve already reviewed and saved for some reason, or those I still want to read. Perhaps categories such as “read if bored on the train”, “use as reference in grant proposal”, “write about in blog post” would be more effective.

And those categories are actually something I will try to implement this year! The last one in particular should be interesting: I really dread organizing my favorites, and I find it difficult to decide on blog post topics — so, why not try an approach that has already worked for me elsewhere and kill two birds with one stone? I just need to decide on the place – ShareLateX does not seem really appropriate this time. Don’t forget to check in later to see the results!

Stable detection of abnormalities in medical scans

This is my second post about a paper I wrote. This time it is about Label stability in multiple instance learning, published at MICCAI 2015. Here you can see a short spotlight presentation from the conference. The paper focuses is on a particular type of algorithms, which are able to detect abnormalities in medical scans, and a potential problem with such algorithms.

What if I told you that I like words “cat”, “bad” and “that”, but I don’t like the words “dog”, “good” and “this”. Did you spot a pattern? To give you a hint, there is one letter that I like in particular*, and I don’t like any words that don’t have that letter. Note that now you are able to do two things: to tell whether I’d like any other word from the dictionary, AND why that is.

You are probably asking yourself what this has to do with medical scans. Multiple instance learning (MIL) algorithms out there that are faced with a similar puzzle: each of the scans in group 1 (words that I like) has abnormalities (a particular letter), and each of the scans in group 2 is healthy. How can we tell whether a scan has abnormalities or not, and where those abnormalities are? If the algorithm figures this out, it could detect abnormalities in images it hasn’t seen before. This is a toy example of how the output could look like:

lungs1

Detecting whether there are any abnormalities is slightly easier than finding their locations. For example, imagine that the scan above has the lung disease COPD, and therefore contains abnormalities. Now imagine, that our algorithm output has changed:

lungs2

The algorithm still correctly detects that the scan contains abnormalities, but the locations are different. If the locations of the abnormalities were clinically relevant, this would be a problem!

Of course, in the ideal case we would evaluate such algorithms on scans, where the regions with abnormalities are manually annotated by experts. But the problem is that we don’t always have such annotations – otherwise we probably would need not to use MIL algorithms. Therefore, the algorithms are often evaluated on whether they have detected ANY abnormalities or not.

In the paper I examined whether we can say something more about the algorithm’s output, without having the ground truth. For example, we would expect a good algorithm to be stable: for one image, slightly different versions of the algorithm should detect the same abnormalities. My experiments showed that an algorithm that is good at detecting whether there are any abnormalities, isn’t necessarily stable. Here is an example:

copd stability

Here I compare different algorithms – represented by different colored points – on a task of detecting COPD in chest CT scans. The y-axis measures how good an algorithm is at detecting COPD (whether there are any abnormalities) – the higher this value, the better. Typically this would be the measure of how researchers in my field would choose the “best” algorithm.

I proposed to also examine the quantity on the x-axis, which measures the stability of the detections: a value of 0.5 means that multiple slightly different versions of the same algorithm only agree on 50% of the abnormalities they detected. Now we can see that the algorithm with the highest performance (green square) isn’t the most stable one. If the locations of the abnormalities are clinically relevant, it might be a good idea to sacrifice a little bit of the performance by choosing a more stable algorithm (blue circle).

A more general conclusion is: think carefully whether your algorithm evaluation measure really reflects what you want the algorithm to do.

%d bloggers like this: