So, it took me a bit longer to get back to this than I originally thought, but time flies when you’re having fun. My previous post on transitioning from the academia to industry received a lot more positive feedback than what I had hoped — many people reached out to me afterwards, some to ask questions and some to empathize with the feelings I was discussing. Thus it felt prudent to stay true to my word and provide the second part of this nascent trilogy by telling how it’s going after a few months. In short: working in the industry rocks. For details: keep reading.
This text will consist of 4 parts, which are “How has it been in general”, “What do I actually do day to day?”, “How does the private sector compare to the Academia?” and “Afterword”. I’ll try to keep the descriptions as general as I can, both to make the text more accessible to people planning to work in an non-identical sector and to make sure I stay well within the various NDAs I have signed.
Before starting the story proper, this piece of text requires an even bigger caveat than the one before. In addendum to the previous small print, here in particular my experiences are strongly tied to the (type of) company I ended up working in. YMMV.
How has it been in general?
In short; I really do like it. To elaborate a bit, I haven’t regretted the decision to transition at all so far. There are many things that I do miss in varying amounts, but they do not counter all the positive things — not even close.
The work I do is challenging enough that I have to use the skill sets of a researcher. The problems might not be quite at the level of “Nobody has been able to do this and brilliant minds have been trying for decades.” but still in the level of “No one has done anything quite like this before, and many of the relevant projects are company secrets that we can’t get direct access to.” In particular, many of the projects start with studying the research literature on the topic on how these types of problems have been approached by other groups, planning a first approach and starting research-type cycle of “try, fail at least partially, learn from failure, adjust and start again”.
The problems can also be very open-ended, where the task might be improve the profitability of a system by reducing computational resource expenditure, improving accuracy or both. Sometimes the solution might even be “we don’t actually need to run this complicated selection component, we get a high-enough accuracy just by always choosing the last element in the list anyway”. This leaves a lot of room to maneuver and thus room for creativity and out-of-the-box thinking. What I have especially appreciated (and trepidated) from the start is that I’ve been given at the same time both big responsibilities and quite a lot of freedom in how to perform them.
There is still a slight difference between my daily work and doing research in math, though. Every now and then when I do end up doing some actual research-level mathematics, usually when a collaborator calls with a cool idea or question, I am reminded of the contrast between being able to do good work on a machine learning project and doing something in which I have perfected my skills for a decade. So it’s not quite the same, and sometimes I do feel the ping of yearning, but this does not happen daily or even weekly. Also, the fact of the matter is that I do have time outside the nine-to-five schedule that I could use for research questions, but I mostly don’t; in increasing amounts it goes to spending time with the family or to various hobbies. On average I think I spend a few hours every month thinking again about one of the toy problems I like to play with, and a varying amount of time to discuss with some of my co-authors on some referee-reports or new ideas related to finished projects. I think that a big factor here is that I get to learn and do things on the job which are complicated enough to satisfy that original research itch for now, but we’ll see what I have to report in a few years.
To summarize, despite the notable difference to my previous life in math research in the content and some differences in the spirit of things I do, I haven’t had an identity crisis. This is something I had been a bit worried about before, since after doing mathematics for well over a decade my self-image is/was largely built around the idea of being a mathematician. Relying on the old adage “if it works, don’t touch it” I haven’t spent too much time probing my concept of self recently, but it seems that my mind is either happy in identifying as “someone who solves complicated technical problems and then explains them” (which is almost the same as saying that you’re a mathematician) or then I still find myself as a mathematician who’s just doing some more applied stuff at the moment. It might be even possible that I exist independently of the label I put on the stuff I do, who knows.
What do I actually do day to day?
First of all, to set the scene, I work as a part of the AI-team at Digital Workforce Services, which has the delightful balance of being like a small startup inside an established company. This means that the workplace in general is secure, but within the AI-team there are not so many dedicated responsibilities or large bureaucracy; anyone can, at least in principle, get to do anything. To my mind this has mostly ups as we get to experience many parts of the project workflow and I’ve been getting to learn various new things, gravitating towards responsibilities where I can excel.
My actual job title is ‘Data Scientist’, which by itself does not tell very much, since ‘Data Scientist’ is kind of an umbrella term that can describe various different roles going from machine learning engineers to various consultancy-type roles. In my particular case I do a bit of everything, and the main tasks I tend to work on are:
- Writing Python code to implement various machine learning and non- machine learning solutions to technical problems.
- Discussing with the project team and planning what we should do and how.
- Discussing with managers, salespeople and customers on how the current projects are going.
- Discussing with managers, salespeople and customers about what we could provide in the future.
Note that three out of four of my main tasks start with the word ‘Discussing’, and even programming usually includes chats with various colleagues. So the part about ‘communication skills being important’ mentioned in the previous part was more than just words.
On a monthly average I’d say that my time is split roughly 50/50 between the computer interaction part of my job, the coding, and the human interaction part of my job, the meetings. Let’s look at these separately.
The coding
The supermajority of the technical side of my daily functions is writing code and Python is the language of choice for our team. Before starting here, most of my programming experience was in fact with Python, but that is not saying much as my previous projects were usually small scripts of a few dozen lines to do technical calculations or some graphical presentations of things. Even when doing ‘a lot of coding’ in my postdoc days, this would usually amount to using some hours within the span of a week to produce a just good enough solution to some particular problem. The amount of expertise I’m gaining now when I’m spending several hours every single day with Python is just exhilarating. The Bill Gates quote, “We tend to overestimate what we can do in a day, and underestimate what we can do in a year.” seems to fit my situation very well, even after a few months instead of a year, and I can’t wait to see how my skills look at the one year mark.
Some people have asked me what was the actual level of my Python skills when I was applying for non-academical jobs for the first time. A good example of my then high-level performance might be this visual cryptography thing. It’s not awful, I think, as it is pretty well commented and you can mostly follow what it is doing, but it is far from great or even good. In particular, even though it is doing well the one single thing it’s doing, it would be really hard to extend, maintain or alter. In particular, if you read uncle Bob’s book (which I highly recommend) you can see that my code is breaking basically all the guidelines. The major thing here, from my non-expert’s point of view, is that there is a huge difference between some code you wrote to do a thing on your own computer, and with code that will be worked on by several people and should yield an actual usable and maintainable product. (Another book recommendation on that topic.)
With regards to creating code, I knew beforehand that there are more nuances to the Art than I could even guess, so I wasn’t starting quite at the top of the bad peak in the Dunning-Kruger curve, but I’m still not sure if I’ve started to properly ascend or if I’m still just learning more about how little I know. The academical meta-skill of “how to cope with impostor syndrome” has helped me dramatically while improving myself in this endeavour, but not as much as my new colleagues who seem to have a lot of patience to answer my repeated questions.
Finally — and I guess this might be true of most things — the further I get and the more I learn to use the fancier methods of Python or of the more general machine learning things, the more important it gets to remember when not to use the shiniest and newest tools. There have been cases where we’ve noticed that a simple algorithm can produce a 90% accuracy with 0.1% of the computational resources that our fancy ML system might use to produce 95% accuracy. There is never a singular correct solution, or even a well-defined solution in the mathematical sense, just (hopefully) good solutions with various tradeoffs, which makes the work all the more interesting.
The meetings
I usually spend an hour or two in various kinds of meetings every day, but this number can vary from zero to eight hours depending on the day and week. One excellent thing I have noticed, though, is that unlike in Academia, most meetings are not an (un)avoidable evil but an essential part of various processes. In particular; the meeting is not a boogeyman keeping you from doing the actual work, but instead the meeting is part of the actual work. This also means that the goals and expectations you have do take into account the fact that a large part of your work effort in the project is partaking in meetings. As this is true for all the other people as well, the meetings themselves can be very fruitful events indeed.
The importance and variety of the meetings also increases in a setting where most of the people do not hold a PhD in my particular narrow branch of research. In mathematical research I often took part in projects that I could not have done on my own, but even in those I had a very good grasp on the skill sets, challenges and workload of the other people in the project as we were all researchers in math. In the industry the projects have a much wider scope; not only could I not do them on my own because my focus in a specific skill set is not covering the area, but there are whole domains of expertise that are completely alien to me yet crucial to the project. The projects transition the capabilities of a person and span across various domains, which means that no single person is capable of understanding everything in detail. This necessitates a huge amount of communication between both individuals and groups. Thus any project needs to be organized in various pieces that abstract the inner mechanical workings of those pieces and which then interface on the abstraction levels. In short; meetings are how the interfaces of the various pieces interact to make sure that the necessary amount of information is flowing.
Here I feel that the caveat of this article has to be emphasized, since I’ve probably gotten very lucky with the company I work in, especially regarding the meetings culture. I am vividly aware that not all meetings outside the Academia are beautiful congregations of fluid information sharing, but in my experience they can be positive events in a workday. (Though this might be also affected by the Covid-pandemic and people noticing how many meetings really can be just emails.) I furthermore don’t want to claim that every single meeting I’ve been in has been a positive event; naturally there have been times when I’ve realized that I’ve ended up in a meeting where my input will not be needed nor will I need the information presented, but these are in the clear minority.
Finally, somewhat relatedly, a standard task for us is to respond to Request For Proposals (RFPs) which are basically announcements by possible customers on projects in which they would like companies to bid for solutions. In my eye, these are basically grant applications, as you know what your skills are and what the client would need, and then you need to bridge the gap between these two and explain why your particular skill set is so very suitable for their task. The bids for a RFP naturally require input from various teams in the company, meaning that I get to use my application-writing skills without needing to bog down to the financial details.
How does the private sector compare to the Academia?
I don’t want to start by completely bashing the academy, but I really need to point out that working in the academia is not always fun. The longer I have now spent in the industry, the more it seems to pop out that many researchers seem to have formed working habits that would be illegal in the private sector. For me the reason behind pushing myself to unhealthy limits was that I didn’t feel like I was working for a university as a grad student or postdoc, I was just doing math while someone was giving me money on a monthly basis so that I can pay my bills and buy food. The blurred lines between ‘what I do for living’ and ‘what I am’ can give rise to very satisfying experiences, but it can also easily mean that you overextend and damage yourself by working too much. The following quote from Chesterton feels like it fit me (and many colleagues as well) quite well:
“A man must love a thing very much if he practices it without any hope of fame or money, but even practice it without any hope of doing it well. Such a man must love the toils of the work more than any other man can love the rewards of it.”
You don’t care that you work for 60+ hours a week and never rest properly in an environment where you have continuous uncertainties about your future when you can’t see the difference between your work and yourself. Even with a supporting family, good friends and exciting hobbies the line between work and non-work can sometimes be hard to find. Don’t get me wrong, I do love the people in the academia, and what I miss most from the academia are the people and communities, but as a whole good people with good intentions can form systems that are not globally optimal for well-being. At the industry side my position is permanent, my pay is higher, my hours are fewer, the work-life balance exists and the goals are reasonable.
After I wrote the first part several people reached out to me to share their feelings with me privately. Many just wanted to talk about various topics, like about how to decide or prepare to transition, but some of them really wanted and needed a ‘safe’ person to talk to as they were worried to share their academia-related insecurities, frustrations and doubts in public. The reason for their worries was that even though in one sense the infamous anonymous review panels in academia are very fair as the people in the panels can tell their opinions without fearing the politics, they do have the downside that people applying for positions can never know who in the coffee room might be deciding their fate in some later occasion. In particular, panelists can judge applicants based on non-transparent criterion like “I do remember this person having doubts about staying in the Academia — maybe we should fund someone else who never expresses such doubts and toes the party line a bit better”. On the one hand you naturally want to hire the person who will stay for sure, but on the other hand the environment you create by punishing people who express any doubts towards the system is not a good one. I’m not saying that this actually happens, but it is enough that people are concerned that it might happen — I know I was, especially after some older and politically more savvy colleagues explicitly warned me about this.
So, after saying that I don’t want to overtly complain about the academy and then writing an essay on the topic, let’s move on to looking at positive things I’ve felt since the transition.
The particularly good things and some of the downsides in the industry contra academia
I’ve already mentioned the immediate, obvious and materialistic aspects: job security, salary, work-life balance etc. Let’s next look at the ones that are harder to measure in numbers.
First of all, the rhythm of life in our company is quite different than in academia. When I was still a postdoc, my duties were basically divided between teaching and research. The teaching responsibilities created daily and weekly deadlines (lecture preparations and lectures etc) with an underlying deadline of getting the full course completed in the x months it would last. Feedback was provided in small form almost weekly and in a more organized form after the course ended. The research parts, instead, were usually concerned with producing research items a few times a year. By the nature of research, this meant that there were no clear deadlines that could be guaranteed. Feedback on research comes quite sporadically and in different forms, and it’s sometimes impossible to guarantee that by working really hard the whole month you’ll get some information on if the idea is good or bad.
In the industry, at least in our company, trying to tell the manager or project leader that “this should be done in about 9–12 months” would not be well received. It is much preferable to spend time, usually together, to figure out how that 9–12 months would be divided — first to major parts like ‘planning’, ‘design’, ‘implementation’, ‘production’, ‘handover’ etc and each of those to smaller parts as well until we reach the scale of 1–10 days. Even though this planning does take effort, it gives various benefits to everyone involved:
- The management, HR and finance people can better estimate the amount of work that is needed, whose input is needed and when, and how much resources will all of this take.
- The technical people will have clear weekly or monthly goals, which means that they (we) can concentrate on reasonable sized blocks of work, get feelings of achievement and not be overwhelmed by the totality of things we need to still accomplish. In addition you get much more frequent feedback
- Everyone involved can try to see which parts are taking roughly the amount of time we were estimating them to take, or if there are certain parts which are falling behind and might require extra resources or redesign.
This goal-orientation is well implemented in our company; you have your goals and if they are achieved then everyone is happy and you get a (monetary) pat on the back. If there seem to be problems in getting things done on time, the ‘system’ detects this and we can figure out what should be changed. The goal-oriented approach also has the nice effect that I get to choose how I wish to get things done — no one but me is managing my day-to-day doings, though we do usually have bi-weekly meetings with the team where we talk about how our projects are going.
Naturally, I have less choice on what I do day to day in the industry than I had when I was a grant-funded researcher. Back then I had the freedom to do what I wanted and fail as I wanted without anyone officially interjecting. Here the projects are born out of customers’ needs and might be put on hiatus or cancelled based on completely external factors. This means both that I don’t have to spend my time wondering which projects I should pursue and for how long, but on the flip side even the most fascinating project I have might suddenly go away, regardless of my personal wishes or performance. In this, as well as in many other aspects of the world, there is a balance to be found between freedom and security. For me personally the tradeoff so far has been great and has done wonders for my mental well-being; having more shorter-term goals that can be achieved — or even exceeded — feels great. (And in hindsight maybe one of the things I sometimes miss from being an undergraduate is having small clear goals i.e. courses that I should pass with good marks and then be done with them. A similar thing holds for teaching.) And even though it can be very annoying to be told to stop working on an interesting project, this is usually a delay instead of cancellation as the company naturally wants to take advantage of the ideas and experience gathered so far in the project.
As an additional but somewhat separate note, an unexpected benefit in industry has been to learn that MS office and other non-open source software is surprisingly good. I used to be an avid user of open source software and Linux, which worked great in an environment where more than half of my colleagues use their own exotic combination of Linux-distros and text-editors for their workflow. But now that 99% of the people surrounding me use the same MS Office package that integrates calendars and workflows, it turns out that these proprietary programs really do seem to have more money available for development etc.
Finally, I mentioned earlier that what I do miss from the Academia are the people and communities, which is true, but I should also emphasize that I have naturally found new great people and communities as well. The difference is that e.g. in the AI-team even though around half of the people hold PhD’s in various complicated topics, we all excel in slightly different fields. We do have a common ground, but everyone in the team is clearly much better than everyone else in at least one large field. A similar thing naturally holds in an even greater extent with regards to colleagues working in other teams. As an experience, it’s not better or worse, just different.
Afterword
Even though this second part was delayed, I’m still planning to turn this into a trilogy, meaning that I’ll hopefully post a third and final part where I describe the feelings after 1–2 years of experience in the industry. If you just can’t wait, someone else also wrote about how the private sector feels seven years after the transition from the Academia. It was a nice read.
To conclude: join the dark side, we have cookies (and job security)!
Acknowledgements:
Thanks to Anne, Olli and an assortment of various Franks on the feedback, their comments improved both this (and the previous) text considerably. Also thanks to all of you who read the previous part, especially to those who reached out.
