<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="https://adivekar.com/feed.xml" rel="self" type="application/atom+xml" /><link href="https://adivekar.com/" rel="alternate" type="text/html" /><updated>2026-11-15T13:33:00-08:00</updated><id>https://adivekar.com/feed.xml</id><title type="html">Abhishek Divekar</title><subtitle>Abhishek Divekar is an AI researcher at Amazon working on LLM evaluation (LLM-as-a-Judge), synthetic data generation, red-teaming and guarding LLMs, and multimodal agents for science. First-author papers at NeurIPS 2026, CIKM 2026, ACL 2026, AAAI 2026 and EMNLP 2024. Chair of the IOAI² AI Models Track and member of the International Scientific Committee of the International Olympiad in Artificial Intelligence (IOAI 2026); Team Leader of Team India at IOAI 2025 (3 Gold, 2 Silver, 1 Bronze, 1 Honorable Mention).</subtitle><author><name>Abhishek Divekar</name></author><entry><title type="html">Setting Up JupyterHub for Your AI Research Team on a New Linux Machine</title><link href="https://adivekar.com/writing/jupyterhub-for-ai-research-teams/" rel="alternate" type="text/html" title="Setting Up JupyterHub for Your AI Research Team on a New Linux Machine" /><published>2025-10-25T00:00:00-07:00</published><updated>2025-10-25T00:00:00-07:00</updated><id>https://adivekar.com/writing/jupyterhub-for-ai-research-teams</id><content type="html" xml:base="https://adivekar.com/writing/jupyterhub-for-ai-research-teams/"><![CDATA[<p class="post-origin">First published as a <a href="https://gist.github.com/adivekar-utexas/229c508a735433b32524900ba4386f34">GitHub gist</a> on 25 October 2025 and updated in February 2026.</p>

<h2 id="the-situation">The Situation</h2>

<p>Suppose you’ve got a team of 2+ people who are working on an AI project (research, business, etc). and they need a common Jupyter machine where they can run notebooks.
You have two common options:</p>
<ul>
  <li>A common JupyterLab: this has the issue that everyone uses the same UI, leading to conflicts. Example: Amazon Sagemaker Notebooks.</li>
  <li>A JupyterHub install: this needs a bit more setup, but gives much better separation.</li>
</ul>

<p>The rest of this post is going to discuss how to setup JupyterHub in a way that:</p>
<ol>
  <li>Everyone gets their user on the machine and their  own home directory, which only they have permissions to.</li>
  <li>No one can access any one else’s home directories or files.</li>
  <li>Everyone has a shared set of conda environments which are read-and-executable.
    <ul>
      <li>These appear for everyone on the JupyterLab interface in the “Select Kernel” button.</li>
      <li>Doing a pip install in any of these environments will override only the user’s environment, no one else is affected.</li>
    </ul>
  </li>
  <li>Everyone can create their own conda envs (only visible to them, do not appear in other people’s list of kernels).</li>
</ol>

<p>Assumptions:</p>
<ul>
  <li>You have provisioned some sort of always-on Linux machine (e.g. an AWS EC2 instance)</li>
  <li>You can SSH into this machine, and the machine has internet access.</li>
  <li>You have <code class="language-plaintext highlighter-rouge">sudo</code> access to this machine.</li>
  <li>This machine has a default user like <code class="language-plaintext highlighter-rouge">ubuntu</code> or <code class="language-plaintext highlighter-rouge">ec2-user</code>.</li>
</ul>

<hr />

<h2 id="the-guide">The Guide</h2>

<p>For the rest of this guide, I will assume the default user is <code class="language-plaintext highlighter-rouge">ubuntu</code>. You can copy-paste these steps and replace it with <code class="language-plaintext highlighter-rouge">ec2-user</code>, etc.</p>

<p>IMPORTANT: DO NOT INSTALL MINICONDA IN THE ROOT USER’S HOME DIRECTORY i.e. <code class="language-plaintext highlighter-rouge">/root</code></p>
<ul>
  <li>A later step in this guide requires you to give access to users to execute a file in the conda environment to spawn their JupyterLab servers. That cannot be done if the environment lies within <code class="language-plaintext highlighter-rouge">/root</code>.</li>
</ul>

<p>When you are logged in as the default user, the prefix on the command line should be something like this: <code class="language-plaintext highlighter-rouge">ubuntu@my-server:~$ </code></p>

<h2 id="step-1-install-conda-in-the-default-users-home-directory">Step 1: Install Conda in the <code class="language-plaintext highlighter-rouge">default</code> user’s home directory.</h2>

<p>If you have already installed conda previously, you can skip this step. Make sure conda was NOT installed within the <code class="language-plaintext highlighter-rouge">/root</code> folder, by running <code class="language-plaintext highlighter-rouge">which conda</code></p>

<p>The following commands will install conda in <code class="language-plaintext highlighter-rouge">/home/ubuntu/miniconda3/</code></p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm ~/miniconda3/miniconda.sh
</code></pre></div></div>
<p>Note: if these commands don’t work, <a href="https://www.anaconda.com/docs/getting-started/miniconda/install#linux-terminal-installer">check out the official page</a> for the latest steps (look for “Linux Terminal installer”).</p>

<p>Next do:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~/miniconda3/bin/conda init bash  ## Or zsh, or whatever shell you use
</code></pre></div></div>

<h2 id="step-2-give-permissions-to-access-conda-environments">Step 2: Give permissions to access conda environments</h2>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo chmod o+rx /home/ubuntu
sudo chmod o+rx /home/ubuntu/miniconda3
</code></pre></div></div>

<h2 id="step-3-setup-the-jupyter_hub_root-dir-and-subdirs">Step 3: Setup the <code class="language-plaintext highlighter-rouge">JUPYTER_HUB_ROOT</code> dir and subdirs</h2>

<h4 id="selecting-the-right-jupyter_hub_root">Selecting the right <code class="language-plaintext highlighter-rouge">JUPYTER_HUB_ROOT</code></h4>

<p><code class="language-plaintext highlighter-rouge">JUPYTER_HUB_ROOT</code> is the root dir of your JupyterHub config and user data.</p>
<ul>
  <li><code class="language-plaintext highlighter-rouge">JUPYTER_HUB_ROOT/config</code> will store JupyterHub <code class="language-plaintext highlighter-rouge">config.py</code> file and the sqlite files for the user passwords etc.</li>
  <li><code class="language-plaintext highlighter-rouge">JUPYTER_HUB_ROOT/user</code> will store all user data (notebooks, files they download, non-shared conds envs, etc).</li>
</ul>

<p>There are two options of where you want to set it up:</p>
<ul>
  <li>You can setup <code class="language-plaintext highlighter-rouge">JUPYTER_HUB_ROOT</code> to be a folder like <code class="language-plaintext highlighter-rouge">/jupyterhub</code>.
    <ul>
      <li>IMPORTANT: do NOT create <code class="language-plaintext highlighter-rouge">JUPYTER_HUB_ROOT</code> inside <code class="language-plaintext highlighter-rouge">/root</code>. Users will not be able to access this folder</li>
    </ul>
  </li>
  <li>(Recommeneded) <code class="language-plaintext highlighter-rouge">JUPYTER_HUB_ROOT</code> should live on a persistent file-like storage like Amazon EFS or any other Network File System (NFS).
    <ul>
      <li>(Faliure Pattern#1) Users tend to download huge files without caring about space. This can easily fill up your entire disk for a fixed-size disk. This blocks everyone’s work since they can’t create files anymore. On an EFS/NFS, the disk size grows basically</li>
      <li>(Faliure Pattern#2) A very common failure is that your machine goes down because someone started a job that uses 100% of RAM and 100% of CPUs, so no one can SSH into the machine anymore or open JupyterLab. In this situation, you have to restart your machine, or even provision a new one. It’s a pain to get the user data out of the machine and into a new one. With an EFS/NFS, you can just reattach it to the new machine and restart JupyterHub.</li>
      <li>(Caveat#1) If you a using an EFS/NFS, you need to make sure that it is mounted to the machine before starting JupyterHub. This is something you need to do manually each time you restart the machine. A good pattern is to use a script to mount the EFS/NFS at boot time. If storing the <code class="language-plaintext highlighter-rouge">JUPYTER_HUB_ROOT</code> on the EFS/NFS, you will never forget to mount because the config file is also on the EFS/NFS.</li>
      <li>(Caveat#2) Amazon EFS is interesting because it does not use traditional usernames and group names. Instead, it uses a unique ID for each user. This is handled automatically in the JupyterHub config file below: each username is mapped to a unique ID, which is the hash of the username. So it will be consistent across machines.</li>
    </ul>
  </li>
</ul>

<p>It is a pain to migrate user data later. Avoid it if you can by using an EFS/NFS, which you can quickly reattach to a new machine.
If you know you will never use an EFS/NFS, you can create <code class="language-plaintext highlighter-rouge">JUPYTER_HUB_ROOT</code> on <code class="language-plaintext highlighter-rouge">/jupyterhub</code> or any common location (avoid <code class="language-plaintext highlighter-rouge">/home/ubuntu/</code>, since it’s weird to nest home directories of users within the default user’s home directory).</p>

<h4 id="creating-the-subdirs">Creating the subdirs</h4>
<p>Let’s assume for simplicity you have selected <code class="language-plaintext highlighter-rouge">/jupyterhub</code> as the <code class="language-plaintext highlighter-rouge">JUPYTER_HUB_ROOT</code>.
Run:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo -i 
sudo mkdir -p /jupyterhub/config
sudo mkdir -p /jupyterhub/user
sudo chown root:root /jupyterhub /jupyterhub/user
sudo chmod 740 -R /jupyterhub
sudo chmod 755 /jupyterhub /jupyterhub/user
</code></pre></div></div>

<h2 id="step-4-install-the-jupyterhub-conda-environment">Step 4: Install the JupyterHub conda environment</h2>

<p>First we sudo:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo -i
source /home/ubuntu/.bashrc
</code></pre></div></div>
<p>You should now see a prefix like <code class="language-plaintext highlighter-rouge">(base) root@my-server:~#</code> in your command line.
Next, run:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>conda create -n jupyterhub python=3.12 --yes  ## Update the Python version to the one you want to use
conda activate jupyterhub
conda install -c conda-forge jupyterhub --yes
conda install jupyterlab --yes
conda install notebook --yes
pip install uv
pip install ipykernel jupyterhub-firstuseauthenticator jupyter-server-proxy
pip install ipython ipykernel ipywidgets
</code></pre></div></div>
<p>Next, give users permissions to execute the following file:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo chmod o+rx /home/ubuntu
sudo chmod o+rx /home/ubuntu/miniconda3
sudo chmod -R o+rx /home/ubuntu/miniconda3/envs/jupyterhub
sudo chmod a+rx /home/ubuntu/miniconda3/envs/jupyterhub/bin/jupyterhub-singleuser
</code></pre></div></div>

<h2 id="step-5-create-the-jupyterhub-config-file">Step 5: Create the JupyterHub config file</h2>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo nano /jupyterhub/config/jupyterhub_config.py
</code></pre></div></div>

<p>Use the following config settings. Important changes to make:</p>
<ul>
  <li>Update <code class="language-plaintext highlighter-rouge">JUPYTER_HUB_ROOT</code> to the path you selected in Step 3</li>
  <li><code class="language-plaintext highlighter-rouge">c.JupyterHub.port = 8888</code>: set this to the port you want to use for JupyterHub. IMPORTANT: this port must be exposed to your users.</li>
  <li><code class="language-plaintext highlighter-rouge">c.JupyterHub.admin_users</code>: set this to the admin users you want to give access to JupyterHub.
Adjust other settings as needed.</li>
</ul>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>c = get_config()

from subprocess import check_call
from hashlib import sha256
import pathlib, re, os, shutil, io

JUPYTER_HUB_ROOT = '/jupyterhub'  ## Update this to the path you selected in Step 3
USER_DIR_PATH = f'{JUPYTER_HUB_ROOT}/user'


def pre_spawn_hook(spawner):
    """Create user:"""
    username = spawner.user.name
    username = username.strip()
    if username in ['root', 'ec2-user', 'ubuntu']:
        raise ValueError(f'Cannot create user "{username}"')
    uname_exp = re.compile("^([a-zA-Z0-9-]+)$")
    if uname_exp.match(username) is None:
        raise ValueError(f'"{username}" is an invalid username. Only alphanumeric characters and hyphens are allowed.')
    uid = int(sha256(str(username).encode('utf8')).hexdigest(), 16) % (4294967294-65536) + 65536  ## Ref: https://unix.stackexchange.com/a/685943
    try:
        check_call(['useradd', '-u', f'{uid}', '-ms', '/bin/bash', '-d', f'{USER_DIR_PATH}/{username}', username])
    except Exception as e:
        print(f'{e}')
    """Create directory:"""
    username = spawner.user.name  # get the username
    volume_path = os.path.join(USER_DIR_PATH, username)
    if not os.path.exists(volume_path):
        os.mkdir(volume_path, 0o755)
    os.system(f'chown {username} {USER_DIR_PATH}/{username}/')

c.Spawner.pre_spawn_hook = pre_spawn_hook
c.Spawner.notebook_dir = USER_DIR_PATH + '/{username}/'

c.JupyterHub.admin_users = set(["admin"])  ## Update this to the admin users you want to give access to JupyterHub.
c.JupyterHub.port = 8888
c.Spawner.args = [
    '--allow-root',
]
c.ServerApp.shutdown_no_activity_timeout = 7 * 24 * 60 * 60
c.MappingKernelManager.cull_idle_timeout = 7 * 24 * 60 * 60  ## Ref: https://www.ibm.com/docs/en/spectrum-conductor/2.4.0?topic=notebooks-kernel-culling-jupyter

c.JupyterHub.authenticator_class = 'firstuseauthenticator.FirstUseAuthenticator'

c.Spawner.mem_guarantee = '2G'
c.Spawner.cpu_guarantee = 1
c.Spawner.default_url = '/lab'  ##Ref https://github.com/jupyterhub/jupyterhub/issues/2603#issuecomm$
c.Spawner.mem_limit = '200G'
c.Spawner.cpu_limit = 40
</code></pre></div></div>

<h2 id="step-6-start-jupyterhub">Step 6: Start JupyterHub</h2>

<p>It’s best to start JupyterHub within screen or tmux, so that it continues running even if you log out.
Here is an example using <code class="language-plaintext highlighter-rouge">screen</code>:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>screen -S jupyterhub
sudo -i
source /home/ubuntu/.bashrc
conda activate jupyterhub
jupyterhub -f /jupyterhub/config/jupyterhub_config.py
</code></pre></div></div>
<p>To logout of screen while keeping JupyterHub running, hold down <code class="language-plaintext highlighter-rouge">Ctrl</code>, then press <code class="language-plaintext highlighter-rouge">a</code> followed by <code class="language-plaintext highlighter-rouge">d</code> (both must be pressed while holding down <code class="language-plaintext highlighter-rouge">Ctrl</code>).</p>
<ul>
  <li>IMPORTANT: On Mac, this is Control key (^) not the Command key (⌘).</li>
</ul>

<p>To test if JupyterHub is working, visit <code class="language-plaintext highlighter-rouge">http://&lt;your-server-ip&gt;:&lt;port&gt;</code>, where <code class="language-plaintext highlighter-rouge">&lt;your-server-ip&gt;</code> is the IP address of your server and <code class="language-plaintext highlighter-rouge">&lt;port&gt;</code> is the port you set in Step 5 (default is 8888 from the config file above).
You should see the JupyterHub login page with a username and password field.</p>

<h2 id="step-7-create-the-admin-user">Step 7: Create the admin user</h2>
<p>It is very critical to create the <code class="language-plaintext highlighter-rouge">admin</code> user first and set its password. This user will have access to JupyterHub admin panel and can stop/restart other users’ JupyterLab servers.
If you don’t do this, and share the JupyterHub login page with your users, someone else might login as the <code class="language-plaintext highlighter-rouge">admin</code>. 
To create the admin user, login with the username <code class="language-plaintext highlighter-rouge">admin</code> and a newly-chosen password.</p>

<p>You can access the admin panel at: <code class="language-plaintext highlighter-rouge">http://&lt;your-server-ip&gt;:&lt;port&gt;/hub/admin</code></p>

<h2 id="repeatable-step-8-create-a-regular-user">(Repeatable) Step 8: Create a regular user</h2>
<p>To create a user, login with the username and a newly-created password. Note that usernames are limited to alphanumeric characters and hyphens.
Warning: at the moment, the config does not restrict the number of users you can create.</p>

<h2 id="repeatable-step-9-create-a-new-conda-environment-visible-to-all-users">(Repeatable) Step 9: Create a new conda environment (visible to all users)</h2>
<p>To create a new conda environment visible to all users in the JupyterLab interface, run the following commands:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo -i
source /home/ubuntu/.bashrc
conda create -n common_env python=3.12 --yes  ## Can use any Python version you want
conda activate common_env
conda install &lt;any other conda packages you want to install&gt;
pip install uv  ## For fast installs of Pip packages
uv pip install ipython ipykernel ipywidgets
uv pip install &lt;any other pip packages you want to install&gt;
python -m ipykernel install --name=common_env  ## Adds the environment to everyone's JupyterLab interface
</code></pre></div></div>

<h2 id="repeatable-step-10-create-a-new-conda-environment-visible-only-to-your-user">(Repeatable) Step 10: Create a new conda environment (visible only to your user)</h2>
<p>This is a step to be followed by each user after they login to their own JupyterLab instance.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>source /home/ubuntu/.bashrc
conda create -n my_private_env python=3.12 --yes  ## Can use any Python version you want
conda activate my_private_env
conda install &lt;any other conda packages you want to install&gt;
pip install uv  ## For fast installs of Pip packages
uv pip install ipython ipykernel ipywidgets
uv pip install &lt;any other pip packages you want to install&gt;
</code></pre></div></div>
<p>You can’t install the environment to be visible to other users, since it will be installed locally on your home directory.</p>

<h2 id="repeatable-step-11-override-packages-from-the-global-conda-environment-visible-only-to-your-user">(Repeatable) Step 11: Override packages from the global conda environment (visible only to your user)</h2>
<p>Often, users want to use an existing global conda environment (visible to all users) but install a few packages/package-versions specific to them.
Luckily, this is possible in much the same way as creating a new conda environment visible only to your user:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>source /home/ubuntu/.bashrc
conda activate common_env
uv pip install &lt;any other pip packages overrides you want to install&gt;
</code></pre></div></div>
<p>These overrides will be specific to your user and will NOT be visible to other users. They live within the user’s home directory, not in the shared conda environment in <code class="language-plaintext highlighter-rouge">/home/ubuntu/miniconda3/envs/common_env</code>.</p>

<h2 id="optional-step-12-install-jupyter-ai-on-the-jupyterhub-machine">(Optional) Step 12: Install Jupyter AI on the JupyterHub machine</h2>
<p>Jupyter AI adds AI coding in a notebook interface.
You can install Jupyter AI on the JupyterHub machine using the following command:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo -i
source /home/ubuntu/.bashrc
conda activate jupyterhub
uv pip install "jupyter-ai[all]" langchain-openai langchain-anthropic boto3
uv pip install jupyterhub-ai-gateway
</code></pre></div></div>
<p>Now, install the environment file which will inject keys for all your users:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo install -D -m 600 -o root -g root /dev/null /etc/jupyterhub/llm.env
</code></pre></div></div>
<p>Modify <code class="language-plaintext highlighter-rouge">/etc/jupyterhub/llm.env</code> to add your keys:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>OPENROUTER_API_KEY=&lt;your-openrouter-api-key&gt;
OPENAI_API_KEY=&lt;your-openai-api-key&gt;
ANTHROPIC_API_KEY=&lt;your-anthropic-api-key&gt;
AWS_ACCESS_KEY_ID=&lt;your-aws-access-key-id&gt;
AWS_SECRET_ACCESS_KEY=&lt;your-aws-secret-access-key&gt;
</code></pre></div></div>
<p>Finally, modify the JupyterHub <code class="language-plaintext highlighter-rouge">config.py</code> file to add the following lines at the end:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>from pathlib import Path

def load_env_file(path):
    d = {}
    p = Path(path)
    if p.exists():
        for line in p.read_text().splitlines():
            s = line.strip()
            if s and not s.startswith("#") and "=" in s:
                k, v = s.split("=", 1)
                d[k] = v
    return d

c.Spawner.environment = load_env_file("/etc/jupyterhub/llm.env")
</code></pre></div></div>
<p>Finally, restart JupyterHub to apply the changes:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>screen -S jupyterhub  ## or screen -x jupyterhub if you are reattaching to an existing screen session
sudo -i
source /home/ubuntu/.bashrc
conda activate jupyterhub
jupyterhub -f /jupyterhub/config/config.py
</code></pre></div></div>
<p>Then, your users will be able to use Jupyter AI in their notebooks.
NOTE: for some providers (e.g. OpenRouter), it might not pick the API key from the environment variable automatically. In that case, your users can set it manually in their notebooks.</p>

<h2 id="optional-repeatable-step-13-use-ray-on-the-jupyterhub-machine">(Optional, Repeatable) Step 13: Use Ray on the JupyterHub machine</h2>

<p>Ray is a very popular library for distributed computing, and a great way to help your users scale their computations. Ray takes care of queueing jobs and distributing them across the cluster. It’s particularly good for GPU jobs, as you can specify a fraction of the GPU to be used for each job.
There are two ways to use Ray on the JupyterHub machine.</p>

<h4 id="option-a-connect-to-an-existing-ray-cluster-from-the-jupyterhub-machine">Option A: Connect to an existing Ray cluster from the JupyterHub machine</h4>
<p>Here, you don’t actually have to do anything with JupyterHub except install the Ray package (<code class="language-plaintext highlighter-rouge">uv pip install ray</code>) in one of the conda environments visible to all users. 
You can use Ray as you would normally:</p>
<ul>
  <li>The Ray head node (i.e. the master), you start a Ray cluster as follows:</li>
</ul>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>clear &amp;&amp; ray stop --force &amp;&amp; ray stop --force &amp;&amp; sleep 5 &amp;&amp; mkdir -p /tmp/ray/ &amp;&amp; rm -rf /tmp/ray/* &amp;&amp; rm -rf /tmp/tmp*.partd &amp;&amp; mkdir -p /dev/shm/ray/ &amp;&amp; rm -rf /dev/shm/ray/* &amp;&amp; ray start --head --system-config='{"object_spilling_config":"{\"type\":\"filesystem\",\"params\":{\"directory_path\":\"/dev/shm/ray\"}}"}'
</code></pre></div></div>
<ul>
  <li>From one or more Ray worker nodes, you can connect to the Ray cluster as follows (replacing <code class="language-plaintext highlighter-rouge">&lt;ray-head-node-ip&gt;</code> with the IP address of the Ray head node):</li>
</ul>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>clear &amp;&amp; sleep $(awk 'BEGIN{srand();print int(rand()*(5)) }') &amp;&amp; ray stop --force &amp;&amp; ray stop --force &amp;&amp; mkdir -p /tmp/ray/ &amp;&amp; rm -rf /tmp/ray/* &amp;&amp; rm -rf /tmp/tmp*.partd &amp;&amp; mkdir -p /dev/shm/ray/ &amp;&amp; rm -rf /dev/shm/ray/* &amp;&amp; ray start --address='&lt;ray-head-node-ip&gt;:6379'
</code></pre></div></div>

<p>To connect to the Ray cluster from the JupyterHub machine, you need to first setup a directory within <code class="language-plaintext highlighter-rouge">/tmp/</code> for the client-side Ray connections to use. These files are cleaned up automatically on reboot and periodically.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo -i 
mkdir -p /tmp/ray_client/  ## For client-side Ray connections by your users.
## Makes client files accessible to everyone now and in the future:
chmod 777 -R /tmp/ray_client/
chmod 1777 /tmp/ray_client/
</code></pre></div></div>
<p>Then, you can use the following code in your Jupyter notebook:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">ray</span>
<span class="c1"># from ray.util.dask import enable_dask_on_ray  # Optional: for Dask on Ray   
</span><span class="kn">from</span> <span class="nn">pprint</span> <span class="kn">import</span> <span class="n">pprint</span>
<span class="n">ray</span><span class="p">.</span><span class="n">shutdown</span><span class="p">()</span>  <span class="c1"># Cleans up any pending Ray connections from a kernel restart
</span><span class="n">pprint</span><span class="p">(</span><span class="n">ray</span><span class="p">.</span><span class="n">init</span><span class="p">(</span>
    <span class="n">address</span><span class="o">=</span><span class="s">'ray://&lt;ray-head-node-ip&gt;:6379'</span><span class="p">,</span>
    <span class="n">ignore_reinit_error</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
    <span class="c1"># log_to_driver=True, # Optional: for more verbose logging
</span>    <span class="n">_temp_dir</span><span class="o">=</span><span class="s">'/tmp/ray_client/'</span><span class="p">,</span>
    <span class="n">include_dashboard</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
    <span class="c1"># Optional: if you want to use any local Python modules which are not installed
</span>    <span class="c1"># on the Ray cluster (e.g. custom libraries you have written), you can add them here.
</span>    <span class="c1"># runtime_env={"py_modules": [  
</span>    <span class="c1">#     &lt;any local Python modules you want to use on the Ray cluster&gt;
</span>    <span class="c1"># ]},
</span><span class="p">))</span>
<span class="c1"># enable_dask_on_ray()  #  Optional: for Dask on Ray
</span><span class="n">pprint</span><span class="p">(</span><span class="n">ray</span><span class="p">.</span><span class="n">cluster_resources</span><span class="p">())</span>
</code></pre></div></div>

<h4 id="option-b-start-a-ray-cluster-on-the-jupyterhub-machine">Option B: Start a Ray cluster on the JupyterHub machine</h4>
<p>If you don’t have multiple machines, you can start a Ray cluster on the JupyterHub machine. 
Ray has a splits the machine’s RAM in to object-store and working memory. If your users are using Ray, this can ensure their jobs do not use more than the total machine memory (which can easily bring down the machine, making it non-SSHable). Their jobs will be queued for execution with Ray.
Here, it is recommended to start Ray in sudo mode, so that it gets special permissions to access the entire machine’s RAM.
You need to first setup two directories for Ray to use:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo -i 
mkdir -p /tmp/ray/
mkdir -p /tmp/ray_client/  ## For client-side Ray connections by your users.
## Makes client files accessible to everyone now and in the future:
chmod 777 -R /tmp/ray_client/
chmod 1777 /tmp/ray_client/
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">/tmp/ray/</code> and <code class="language-plaintext highlighter-rouge">/tmp/ray_client/</code> are cleaned up automatically on reboot and periodically.</p>

<p>Then, as a one-time activity, you need to increase the limits on open files or you may get “Too many open files” OS errors:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo -i
## Backup current limits
cp /etc/security/limits.conf /etc/security/limits.conf.backup
## As per https://discuss.ray.io/t/setting-ulimits-on-ec2-instances/590 
## Also see https://superuser.com/a/1027505 about wildcard(*) vs root
echo "* soft nofile 65535" &gt;&gt; /etc/security/limits.conf
echo "* hard nofile 65535" &gt;&gt; /etc/security/limits.conf
echo "root soft nofile 65535" &gt;&gt; /etc/security/limits.conf
echo "root hard nofile 65535" &gt;&gt; /etc/security/limits.conf

echo 'ulimit -n 65535' &gt;&gt; ~/.bashrc  ## OR whatever you use: ~/.zshrc, ~/.bash_profile, etc. This loads them when logging in.
### Logout of sudo, then login to check changes:
exit
sudo -i
ulimit -a  ## "open files" should be 65535
</code></pre></div></div>

<p>(Repeatable) Then you run the following ray command on the JupyterHub machine using sudo:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo -i 
source /home/ubuntu/.bashrc
conda activate common_env_with_ray  ## This conda environment should have ray and ray[client] installed.
clear &amp;&amp; ray stop --force &amp;&amp; ray stop --force &amp;&amp; sleep 5 &amp;&amp; mkdir -p /tmp/ray/ &amp;&amp; rm -rf /tmp/ray/* &amp;&amp; rm -rf /tmp/tmp*.partd &amp;&amp; mkdir -p /dev/shm/ray/ &amp;&amp; rm -rf /dev/shm/ray/* &amp;&amp; ray start --head --system-config='{"object_spilling_config":"{\"type\":\"filesystem\",\"params\":{\"directory_path\":\"/dev/shm/ray\"}}"}'
</code></pre></div></div>
<p>(Repeatable) To connect to this Ray cluster you can use the following code in your Jupyter notebook:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">ray</span>
<span class="c1"># from ray.util.dask import enable_dask_on_ray  # Optional: for Dask on Ray   
</span><span class="kn">from</span> <span class="nn">pprint</span> <span class="kn">import</span> <span class="n">pprint</span>
<span class="n">ray</span><span class="p">.</span><span class="n">shutdown</span><span class="p">()</span>  <span class="c1"># Cleans up any pending Ray connections from a kernel restart
</span><span class="n">pprint</span><span class="p">(</span><span class="n">ray</span><span class="p">.</span><span class="n">init</span><span class="p">(</span>
    <span class="n">address</span><span class="o">=</span><span class="s">'ray://127.0.0.1:10001'</span><span class="p">,</span>
    <span class="n">ignore_reinit_error</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
    <span class="c1"># log_to_driver=True, # Optional: for more verbose logging
</span>    <span class="n">_temp_dir</span><span class="o">=</span><span class="s">'/tmp/ray_client/'</span><span class="p">,</span>
    <span class="n">include_dashboard</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
    <span class="c1"># Optional: if you want to use any local Python modules which are not installed
</span>    <span class="c1"># on the Ray cluster (e.g. custom libraries you have written), you can add them here.
</span>    <span class="c1"># runtime_env={"py_modules": [  
</span>    <span class="c1">#     &lt;any local Python modules you want to use on the Ray cluster&gt;
</span>    <span class="c1"># ]},
</span><span class="p">))</span>
<span class="c1"># enable_dask_on_ray()  #  Optional: for Dask on Ray
</span><span class="n">pprint</span><span class="p">(</span><span class="n">ray</span><span class="p">.</span><span class="n">cluster_resources</span><span class="p">())</span>
</code></pre></div></div>]]></content><author><name>Abhishek Divekar</name></author><category term="engineering" /><category term="infrastructure" /><category term="guide" /><summary type="html"><![CDATA[A step-by-step guide to a shared JupyterHub: per-user home directories, shared read-only conda environments, personal envs, Jupyter AI, and Ray.]]></summary></entry><entry><title type="html">Tips for a Successful Resume and Referral at Amazon</title><link href="https://adivekar.com/writing/tips-for-amazon-referrals/" rel="alternate" type="text/html" title="Tips for a Successful Resume and Referral at Amazon" /><published>2024-09-09T00:00:00-07:00</published><updated>2024-09-09T00:00:00-07:00</updated><id>https://adivekar.com/writing/tips-for-amazon-referrals</id><content type="html" xml:base="https://adivekar.com/writing/tips-for-amazon-referrals/"><![CDATA[<p class="post-origin">First published as a <a href="https://gist.github.com/adivekar-utexas/9e352afca68802cce0d92a66d6b2cc4a">GitHub gist</a> on 9 September 2024 and updated through 2025.</p>

<p>🚨 <strong>(Update June 2025)</strong> 🚨 This little post has gained a huge amount of interest, for which I am very grateful. However, it’s been overwhelming to keep up with 40-50 reachouts per week. To make things manageable, I am selectively focusing on referrals for <strong>full-time</strong> Applied Scientist and Data Scientist roles only. 
The rest of this post still contains tips for other tech roles (Software Engineer, Data Engineer, BIE, etc) and details on the hiring process in general; I hope these are valuable for you. Good luck on your journey!</p>

<hr />

<p>I am always happy to provide referrals for folks applying to Amazon for a variety of roles. 
Amazon is a FAANG, so I know that landing a job there can make a big difference to someone’s career, and I am happy to spend the time to provide a referral.</p>

<p>If you are interested, please read the points below <strong>very carefully</strong> before reaching out (I only accept requests via <a href="https://www.linkedin.com/in/ardivekar/">LinkedIn</a>). This is to save time on both sides 😄 I usually check around 9am EST / 8pm IST. If you reach out, please give me upto 72 hours to revert.</p>

<p><strong>I will not entertain requests from those who have clearly not read these points, regarless of how great your accomplishments are.</strong> No exceptions.</p>

<p>The tips:</p>
<ol>
  <li>
    <p>It is up to you to visit <a href="https://www.amazon.jobs/">amazon.jobs</a> and search for open positions.</p>
  </li>
  <li>‼️ Once you find a job, <strong>DO NOT begin the application for the job yourself</strong>; if you do so, the portal does not allow me to refer you for the same job‼️. <strong>I will not go forward with a candidate who has done this.</strong>
    <ul>
      <li>Some candidates have said that starting the application (but not submitting it) also causes this to happen; if you have started the application for a particular job (even from LinkedIn), it makes sense to finish the application, since no Amazon employee can refer you for the same job anymore.</li>
    </ul>
  </li>
  <li>Pick jobs which <strong>best fit your profile</strong> and were recently posted (ideally, &lt;1 week ago). A 1+ month old job is probably already filled (or has many candidates in the late stage of the hiring pipeline).
    <ul>
      <li>The Basic Qualifications and Preferred qualifications are usually set by managers, and taken seriously by recruiters. <strong>Hiring managers get maybe 100+ applicants for a job, and will typically ignore you if you have not worked in the domain before.</strong> Good role-fit is a huge deal in a successful referral. I will not refer you for roles where there fit is clearly not there. E.g. if you are a stellar embedded engineer, I will not refer you for a Full-Stack role. You should have worked in a similar domain.</li>
      <li>If the role is super generic e.g. “Student Programs” or “University Talent Acquisition”, you should just apply directly…a referral does not carry any extra weight.</li>
    </ul>
  </li>
  <li>
    <p>I can manage upto 5-6 referrals per person, for at most 2 different roles (e.g. Software Engineer and Data Scientist). Beyond this does not make sense.</p>
  </li>
  <li>Below are important rules to decide if a job is right for you. These are my personal rules, created because I receive several requests which are inappropriate for the level/job given the candidate’s background.
    <ul>
      <li>I will NOT refer for non-Tech roles (e.g. Associate, Human Resources, Legal, Sales, Business Analyst, etc). I have no influence in non-Tech roles. If it has the word “Engineer”, “Developer” or “Scientist” in the title, it is a Tech role and I am okay to refer.</li>
      <li>If you are transitioning to Tech or freshly out of undergrad/Masters, look for positions asking for 0-3 years work experience. If it says “1+ year of non-intern experience”, understand that this is a hard requirement by the hiring manager. No exceptions.</li>
      <li>If you’re applying for an entry-level position and you are already working, I will only refer if you have been working for 1+ years (non-internship experience). <strong>No exceptions unless you already work at a FAANG or equivalent company</strong>.</li>
      <li>If you’ve held a software/data science job for 3+ years, you can look for mid-level positions. <strong>No exceptions unless you already work at a FAANG or equivalent company</strong>.</li>
      <li>Multiple referrals <strong>must</strong> be for different teams (i.e. different manager). Ideally, they should be for different departments, since then the recruiters may also be different.</li>
      <li>I will only refer for Applied Scientist on a case-by-case basis (e.g. if you have ML research publications). I am okay to refer for Data Scientist without research experience, <strong>provided</strong> you currently work as a Data Scientist.</li>
      <li>‼️ Once you find a job, <strong>DO NOT apply for the job yourself</strong>; if you do so, the portal does not allow me to refer ‼️ I will not go forward with a candidate who has done this.</li>
    </ul>
  </li>
  <li><strong>Resume</strong>: You should have your resume ready before you DM me. Take care to include the following:
    <ul>
      <li><strong>Must include</strong>: your (a) city (b) zipcode/pincode (c) phone (d) email. These are necessary fields I must enter on the internal referrals portal. I cannot proceed unless your resume has all 4.</li>
      <li>Ordering of resume sections:
        <ul>
          <li>If you are studying, the order should be: Education, Experience, Publications (optional), Skills, Projects (2-4), Extracurriculars (optional).</li>
          <li>If you are working, the order should be: Experience, Education, Publications (optional), Skills, Projects (2-4), Extracurriculars (optional).</li>
          <li>In Education, add your GPA, unless it’s bad (“bad” is &lt;3.2/4.0 or &lt;7.5/10).</li>
          <li>In Education, don’t put anything before high-school (in India terms, don’t put 10th grade, no one cares).</li>
          <li>Experience should be reverse-chronological.</li>
          <li>Mention dates on ALL Education &amp; Experience line-items. In Education, mention “Expected graduation: XX”</li>
        </ul>
      </li>
      <li><strong>Always tailor your resume for the job you are applying to</strong>. If you are applying to different roles (e.g. MLE and Data Scientist), then you should create multiple versions of your resume, since it will be</li>
      <li>It’s <strong>much</strong> better to have a short, 1-page resume where every single point is impressive, rather than a 2-page resume which is 50% fluff; the former demonstrates you know what is actually important.
        <ul>
          <li>At a section-level, keep only the most impactful things you have done which are relevant. It’s better to have 3 strong, impactful bullet-points described in detail, rather than 5 half-hearted points.</li>
          <li><strong>Cut out irrelevant content ruthlessly.</strong> If you were swim captain or led the debate team or general treasurer of XYZ etc, it’s not relevant to the job; remove it. Hackerrank/Leetcode scores are okay to keep for a Software Engineer role, since it is relevant. Kaggle is okay for Scientists but hardly makes a difference.</li>
        </ul>
      </li>
      <li>Add <strong>datapoints</strong> to quantify/qualify your impact.
        <ul>
          <li>This helps you stand out from other candidates, simply because when I read a line like:”solution improved model accuracy from 70% to 85% (1500 bps)” it is objectively impressive and clear that (1) you understand the impact of your work (2) your work is itself impactful.</li>
          <li>You will be very surprised how impactful your work might be, when you actually sit down to measure the impact.</li>
          <li>It’s okay to use back-of-the-envelope calculations here, so long as they are grounded in reality and describe your impact, not your team’s/department’s. <strong>Don’t</strong> make up impact numbers; make the effort to measure them.</li>
          <li>Don’t add irrelevant datapoints. If everyone in Amazon Retail says they work on a platform generating $500 billion per year, it’s meaningless. Focus on impact measures for projects where, you personally, were the main owner.</li>
        </ul>
      </li>
      <li>👨🏻‍🎓 If you are a Masters/PhD student, make sure you include the expected graduation date. For folks in a part-time degree like UT’s MSCS online or GaTech’s OMSCS, please additionally mention that the degree is “Part-time” and that you are available to work full-time.</li>
      <li>🧑🏻‍💼 Unless you have 10+ years of (industry experience + grad studies), your resume should be ONE page long. This is perhaps the biggest thing you can do to optimize your resume.</li>
      <li>When reading resumes, I mentally note when someone’s workex/projects demonstrate they are authentically “ML obsessed” or “SWE obsessed”. This is usually indicated by the fact that they have gone <strong>very deep</strong> into a rabbit hole, upto the limit of what is technically possible, rather than doing 5 generic projects whose code can be found online (e.g. “Hostel booking system”, “YOLO model for object detection”, etc). I like seeing this, as do most interviewers.</li>
      <li>For Scientist roles, add any Publications in a separate section, underlining your name on the author-list (if you lack space, mention “first author”, “co-first author”, “third author”, etc).
        <ul>
          <li>If you don’t mention this detail, I will assume you “breathed on the paper”, i.e. you were in research meetings but didn’t contribute anything substantial.</li>
          <li>If any of your papers is in a pay-to-publish venue, I will NOT refer you. No exceptions.</li>
        </ul>
      </li>
      <li>Include the word “sublime” when your reach out so that I know you have actually read this. I will ignore you otherwise, or just send a link back to this post.</li>
    </ul>
  </li>
  <li>
    <p>You understand that a referral does not guarantee an interview or that the recruiter will reach out. The hiring process is messy at every company, and not in my control.</p>
  </li>
  <li>(Optional) It helps to keep a small blurb of your most impactful achievements ready (3-4 lines). If there’s a role for which your background is a great fit, I am usually happy to email the hiring manager to make sure your profile is surfaced. I will only do this for candidates with 3+ years experience, or those with published research papers.</li>
</ol>]]></content><author><name>Abhishek Divekar</name></author><category term="career" /><category term="hiring" /><summary type="html"><![CDATA[My rules for referrals, and how to write a resume that survives a hiring manager's first read: role fit, impact datapoints, ordering, and one page.]]></summary></entry><entry><title type="html">Doing an MS Thesis While Working Full-Time as an ML Scientist</title><link href="https://adivekar.com/writing/thesis-while-working-full-time/" rel="alternate" type="text/html" title="Doing an MS Thesis While Working Full-Time as an ML Scientist" /><published>2023-12-15T00:00:00-08:00</published><updated>2023-12-15T00:00:00-08:00</updated><id>https://adivekar.com/writing/thesis-while-working-full-time</id><content type="html" xml:base="https://adivekar.com/writing/thesis-while-working-full-time/"><![CDATA[<p class="post-origin">Written in December 2023 for <a href="https://mscshub.com/">MSCS Hub</a>; republished as a <a href="https://gist.github.com/adivekar-utexas/7eb8d78b61de5a12b23dd4dd4b9db9a4">GitHub gist</a> in April 2025, with an update from November 2024.</p>

<p>Background: I work as an ML Scientist at FAANG. I started the UT Austin MSCSO program in Fall’2020, and this review was written in Dec’2023. 
This review is very specific, so a motivated person can probably find me. Let me save you the trouble, here is my LinkedIn: linkedin.com/in/ardivekar/ . Feel free to reach out with questions. I’m usually also on the MSCSO Slack.</p>

<p>I applied for UT Austin’s MSCS Thesis in Dec’2022, and completed the Thesis over 3 semesters (Spring’23 to Spring’24). 
My thesis was in the NLP domain, during 2023 when AI went mainstream via ChatGPT, and new LLMs were released every single week. As you can expect, this makes my experience unique. But I expect everyone’s thesis experience will be unique since it depends on your personal strengths and interests.</p>

<p>Another thing which I did different from other students is that I framed my own research problem. This is recommended only if you’re really dead-set on one idea, like I was. Otherwise it’s okay to go with an idea your Thesis advisor gives you. But make sure you’re eventually doing your own research, not just contributing to someone else’s project.</p>

<p>I’ve written about the process I followed to reach out to MS thesis professors. I’ve tried to put this together in a step-by-step fashion, starting from the very basics: <a href="/files/work-with-professors.pdf">How to reach out to professors for an MS thesis (PDF)</a>.</p>

<p>If you have questions on the steps to find an advisor, please <a href="https://linkedin.com/in/ardivekar/">reach out to me on LinkedIn</a>.</p>

<hr />

<p>This is in response to a question on UT Austin MSCSO Slack #thesis-option, posted in 2024.</p>

<p>Q: “What are the pros and cons of doing a thesis?”.</p>

<p>A: Personally, I’ve grown a lot as a researcher, both in skills and temperament.</p>

<p>I was an industry researcher before the thesis, and had published ML papers internally (at Amazon we have an annual internal conference with ~1k submissions and acceptance rate of ~30%). I felt like I knew what I was doing, and the thesis would be a breeze. But I had never had the perspective of academic research, where you pit your ideas against the very best work. My thesis also focuses on LLMs, where the “best” changed weekly in 2023.</p>

<p>The last year (2023) has been very humbling. I feel like I better understand what NLP research is about, and the challenges. My advisor is very good, but at the end its up to me to do the work and push forward the ideas I think are best. This has led me to learn a lot of skills, like how to read a lot of papers fast, how to strategically cut corners to get useful experimental results, and how to orchestrate code which runs LLMs in parallel on a 100+ GPU cluster.</p>

<p>As to cons? Well, like I said it can feel humbling. The thesis itself is a long grind which does not always produce results.
For example, imagine you and your advisor agree that in 2 weeks, you should get results for one set of experiments. You quickly realize 2 weeks is quite a short time to implement this fairly complex idea. But you have to get it done, so you ignore calls from your spouse, push deadlines at work, and hack till 4am at night. And finally, your experiment runs and you get some results. But they are not good results…not enough to publish.</p>

<p>So, you pivot and try a variation of the experiment. This also does not work. In a panic you read a lot of the research papers, and you get some new ideas but you also get the feeling like people have been doing work which is much more in-depth than your own experiments.</p>

<p>You pivot your approach and try again, and again, for 3 months. At the end, you’ve built an excel table of results from many, many variations of the same core idea. But none of the results are strongly positive, and you aren’t sure why (was it a code bug? data not good enough? issue with analysis metric? so many options…). Your advisor was initially excited about your project, but now seems less excited and is trying not to show it.</p>

<p>At this point, your self-confidence has been steadily taking a beating for 3 months. You feel like your ideas are not good, that you are stupid, and (most importantly) that you’ve wasted time which you and your advisor could have given to something more productive or enjoyable.</p>

<p>This is the reality of doing research. To continue pushing forward, you have to be very motivated by the idea you are working on, or just be bull-headed enough (or desperate enough) that you keep working on it regardless.</p>

<p>At different points in time I have experienced all these feelings. What set me straight was, after one set of failed experiments, I attended ACL conference. ACL is the top NLP conference in the world, and in 2023 it was attended by true experts: I saw Geoff Hinton in-person, delivering the keynote (where he was publicly criticized by Emily Bender). I sat next to Chris Manning at a talk, and rode an elevator with Luke Zettlemoyer. I met a director at FAIR from whom I heard about LLaMa-2 a week before it was launched.</p>

<p>But what stuck with me was the vibe. I did not see the bravado and showmanship you get at tech conferences. Everyone here (especially the students) were aware that they did not know some ultimate truth about NLP; they had just explored an idea, sometimes for years, and found something interesting to show. Some of the presentations were very under-whelming and barely seemed like a contribution, but they were being projected to everyone on a 50-foot screen. Most of the authors were nerds (with various levels of grooming) who stammered occasionally, but were excited about their findings; it was impossible to feel intimidated by them.</p>

<p>I left with the impression that while some of the research I saw was truly very deep, most research is ordinary. And thus, I felt confident that with effort and guidance, I could at least produce “ordinary” work (as per the caliber of a conference like ACL). And that hopefully someday, with enough effort and some luck, I might produce very deep work.</p>

<p>The Thesis is the journey which facilitates that feeling.</p>

<p>EDIT Nov 2024: this story has a happy ending. After 1.5 years of work, the paper resulting from my thesis was accepted at EMNLP 2024 Main (a top-tier NLP venue). You can read the paper here: https://aclanthology.org/2024.emnlp-main.1071/</p>]]></content><author><name>Abhishek Divekar</name></author><category term="research" /><category term="career" /><category term="education" /><summary type="html"><![CDATA[What it was really like to do an NLP thesis at UT Austin during 2023 while working full-time at Amazon, and why I would do it again.]]></summary></entry><entry><title type="html">Amazon Engineer to ML Scientist: Notes on a 2-Year Journey of Growth</title><link href="https://adivekar.com/writing/engineer-to-ml-scientist/" rel="alternate" type="text/html" title="Amazon Engineer to ML Scientist: Notes on a 2-Year Journey of Growth" /><published>2021-12-09T00:00:00-08:00</published><updated>2021-12-09T00:00:00-08:00</updated><id>https://adivekar.com/writing/engineer-to-ml-scientist</id><content type="html" xml:base="https://adivekar.com/writing/engineer-to-ml-scientist/"><![CDATA[<p class="post-origin">Originally published on <a href="https://medium.com/@ardivekar/engineer-to-ml-scientist-notes-on-a-2-year-journey-of-growth-ed4d16d22044">Medium</a> on 9 December 2021.</p>

<p>As of this week, <a href="https://www.linkedin.com/in/ardivekar">I’m officially a Machine Learning Scientist</a> (technically, an Applied Scientist, which is equivalent here at Amazon). This is the culmination of a journey which started ~2 years ago, when I was fortunate enough to join an all-Scientist team at Amazon’s India ML research department, despite being a Software Engineer at the time.
People regularly inquire how I made this move (I don’t have a Masters or a PhD), and they express similar ambitions to work in Applied ML or ML research. This post is my response; it details the key insights and suggestions which helped me realize my goal. If you are considering a similar switch in your career, I hope my words are able to help you. These opinions are my own and do not reflect the views of my employer.
There are four broad aspects which were considerations during my switch: <strong>Personal Investment</strong> , <strong>Developing ML skills, Knowing the Job, and Learning How to be a Scientist.</strong></p>

<hr />

<h2 id="personal-investment">Personal Investment</h2>
<h3 id="recognize-the-opportunity">Recognize the opportunity</h3>
<p>Before moving to the ML department, I had worked as a Software Development Engineer (SDE) for ~2 years. I had a few big launches, and was up for promotion. However, I had grown tired of working in software, when Machine Learning seemed much more interesting, dynamic, and meritocratic.
Around this time, there was an internal opening for a “Research Engineer”: a hybrid between an SDE and Scientist. I reached out. Four interviews later, I was presented an offer…at my current level. This meant I potentially had to throw away ~2 years of career growth as an SDE. After a bout of indecision, I realized that <strong><em>I could always go back to my previous job if I failed</em></strong> ; this convinced me that making the switch was worthwhile.
I will say that I was fortunate enough to not have a family to support or loans to repay; I might have made a different decision in those cases.</p>
<h3 id="have-people-who-support-your-journey">Have people who support your journey</h3>
<p>Outside of my family, my manager Nikhil Rasiwasia and many peers helped me succeed. External validation of your progress is not necessary, but it helps a lot! The journey is long, and the support helps you stay on track.</p>

<hr />

<h2 id="developing-ml-skills">Developing ML Skills</h2>
<h3 id="gain-an-in-depth-understanding-of-machine-learning">Gain an in-depth understanding of Machine Learning</h3>
<p>Take graduate-level courses (Intermediate/Expert on Coursera), actually make notes and solve <strong><em>all</em></strong> the exercises. Be a nerd. This is hard to do when juggling work deadlines, but if you want to be a scientist and not an engineer, there is no replacement.
I personally joined <a href="https://www.cs.utexas.edu/graduate-program/masters-program/online-option/">UT Austin’s part-time Masters program</a>; the fixed syllabus and peer-pressure helped me learn. I’m half-done with the program…compared to when I started, my comprehension of ML fundamentals is much deeper than I thought was possible. <strong><em>Really digging into the math of ML is a game-changer</em></strong>.
Speaking of math, blogs and Reddit posts that discuss “how to break into ML” often repeat the following:</p>
<blockquote>
  <p>“The math you need for ML is Probability, Linear Algebra and Calculus” - Internet
While this is not <strong><em>un</em></strong> true, it’s much more accurate to say:
“The math you need for Machine Learning is 50% Probability, 35% Linear Algebra and 15% Calculus”
Probability really is the language of Machine Learning. <strong><em>Work hard on your probability skills and a lot of the literature becomes much easier to digest</em></strong>. I found “<a href="https://projects.iq.harvard.edu/stat110/home">Introduction to Probability</a>” by Blitzstein and Hwang an extremely approachable introduction to all the major concepts.
At the same time, <strong>you should know how probability and linear algebra intersect</strong> (e.g. Expectation of random vectors) and <strong>how calculus and linear algebra intersect</strong> (e.g. Gradients, Hessians, Jacobians); a lot of the advanced math involves these intersections.</p>
  <h3 id="get-good-at-deep-learning-and-other-popular-tools">Get good at Deep Learning (and other popular tools)</h3>
  <p>If I had to estimate, Deep Learning models are used for ~70% of applied ML projects. It’s not hype any more, and entire departments are organized around this. However, <strong><em>don’t let this be the only ML technique you are familiar with</em></strong>. There are times when a Deep Learning model will lose to a “classic” ML technique like XGBoost, either in terms of ML performance or computational cost; in those cases, you can’t shrug your shoulders and apologize.</p>
  <h3 id="there-are-a-few-topics-every-scientist-is-familiar-with-learn-these-as-soon-as-possible">There are a few topics every Scientist is familiar with; learn these as soon as possible</h3>
  <p>Generally, these topics are part of the syllabus of a graduate-level ML course like <a href="http://cs229.stanford.edu/syllabus.html">Stanford’s CS 229</a>. Books I found useful to learn these topics include:</p>
  <ul>
    <li>“<a href="https://www.cs.huji.ac.il/~shais/UnderstandingMachineLearning/">Understanding Machine Learning: from Theory to Algorithms</a>” by Shalev-Shwartz and Ben-David (Free, medium-difficulty, medium-depth)</li>
    <li>“<a href="https://link.springer.com/book/9780387310732">Pattern Recognition And Machine Learning</a>” by Bishop (medium-difficulty, high-depth)</li>
    <li>“<a href="https://www.cs.ubc.ca/~murphyk/MLbook/">Machine Learning - A Probabilistic Perspective</a>” by Murphy (Free, high-difficulty, high-depth)</li>
    <li>“<a href="https://www.statlearning.com/">An Introduction to Statistical Learning (with Applications in R)</a>” by James, Witten, Hastie and Tibshirani (Free, low-difficulty, low-depth)</li>
    <li>“<a href="https://link.springer.com/book/10.1007/978-0-387-84858-7">The Elements of Statistical Learning</a>” by Hastie, Tibshirani, Friedman (Free, high-difficulty, high-depth)</li>
  </ul>
</blockquote>

<p>For every topic, look among these books (and others) for an explanation which you find most digestible. <strong><em>Write it down in your notes.</em></strong> You can also get a lot out of ML class-notes from university professors, <a href="https://www.stat.cmu.edu/~ryantibs/advmethods/notes/">like this</a>.</p>
<h3 id="conference-papers-are-functionally-similar-to-news-articles-you-will-only-understand-them-if-youre-up-to-date-with-the-story-so-far">Conference papers are functionally similar to news articles; you will only understand them if you’re up-to-date with the story so far.</h3>
<p><strong><em>When introduced to a new area, do not read conference papers first</em></strong> ; instead, start with recent survey papers or (if needed) a good book on the fundamentals of that area. Blogs can be good, particularly if the author is a scientist and not someone learning ML themselves (as these tend to be over-simplified).</p>

<hr />

<h2 id="knowing-the-job">Knowing the Job</h2>
<h3 id="figure-out-what-job-you-really-want">Figure out what job you <em>really</em> want.</h3>
<p>I can’t stress this enough, but <strong><em>to work as a scientist, you should be interested in ML for reasons beyond money and because it’s “cool”</em></strong>. If those are your primary motivations, consider Machine Learning engineering or Data engineering; the pay is similar (higher, sometimes), and you play with ML models without dissecting research papers and running endless experiments. Your career growth will also be more predictable, since it won’t be tied to the whims of peer-reviewers. Cassie Kozyrkov has a <a href="https://towardsdatascience.com/data-sciences-most-misunderstood-hero-2705da366f40">superb article</a> on the competencies of different ML roles, give it a read to figure out which one suits you.</p>
<h3 id="applied-science-is-not-just-research">“Applied” Science is not just research</h3>
<p>Half the job is delivering models into production, and the other half is research. You typically work on projects which are meaningful to the business either immediately or as a long-term investment. The first type are more impactful, while the second type are more “research”-y. Both kinds of projects are important to be a well-rounded Applied Scientist. <strong><em>Attending talks, reading and presenting research papers, reading and presenting business documents, writing code, running experiments, and plotting results forms the bulk of the day-to-day work.</em></strong></p>

<h3 id="learn-from-your-peers">Learn from your peers</h3>
<p>Machine Learning is a vast discipline and no one is an expert (or even familiar) with everything. I noticed my Scientist peers were well-versed with one or two ML domains (NLP, CV, recommendations, etc), but had depth only in a few sub-areas. They also varied very widely in terms of age, and years-of-experience was no real indicator of their level.
Something I found very refreshing was that junior scientists were often asked to present their work, and were treated with respect (this is not very common in academia, as I understand).</p>
<h3 id="be-nice-to-the-engineers">Be nice to the engineers!</h3>
<p>Often, building the model is the most painless part of the project; plugging it into existing systems with the right guarantees of input data, latency, scalability, etc can be extremely challenging. ML/Data engineers are the ones who take ownership of this task; that makes them equally important contributors to the success of an Applied ML project. Treat it as a symbiotic relationship.</p>
<h3 id="pick-your-problems-carefully">Pick your problems carefully</h3>
<p>Some problems can’t be solved with ML. Some problems can be solved by ML, but it’s overkill. Maintaining ML models is a boatload of manual work, as their performance actually degrades if you leave them idle for months or years. If the most effective solve is a piece of code which takes the data and produces an output “prediction”, it’s up to you to communicate this to stakeholders.</p>

<hr />

<h2 id="learning-how-to-be-a-scientist">Learning how to be a Scientist</h2>
<h3 id="differentiate-science-and-engineering-work">Differentiate science and engineering work</h3>
<p><strong><em>Scientific research is extremely different from engineering work</em></strong>. The motivations are different (contribute to common knowledge, vs. business impact), the success criterion are different (paper accepted by a small group of experts, vs. deploy a feature for millions of users), the artifacts are different (8-page PDF you will forget immediately after you publish, vs. code which needs to be maintained for years to come), the usage of prior work is different (must be novel, vs. re-use as much as possible)…the list goes on.
In retrospect, this is an area where I could not self-start. If I was not working with scientists who taught me the ropes, I would probably have gone back to academia to learn how to do high-quality scientific work. I don’t have many suggestions here, except to find a similar environment either in industry or academia. The <a href="https://www.scribbr.com/dissertation/literature-review/">articles on Scribbr</a> are a good starting point if you are brand new to scientific research and need to learn the terminology.</p>
<h3 id="get-a-research-mentor">Get a research mentor</h3>
<p>Every ML domain does research “differently”. There are different benchmark datasets and different standards for publication. For an extreme example, take a look at the differences between accepted papers at <a href="https://icml.cc/Conferences/2021/Schedule?type=Poster">ICML 2021</a>, <a href="https://2021.aclweb.org/program/accept/">ACL 2021</a>, and <a href="http://learningtheory.org/colt2021/accepted.html">COLT 2021</a>, three undoubtedly top-tier conferences in different research areas. ACL leans towards empirical results, whereas COLT is pure theory, and ICML is somewhere in between. All are “Top-tier ML conferences”, but they barely speak the same language.
As an inexperienced researcher, you will not know these idiosyncrasies, and it’s possible to waste months with rejected publications because you didn’t tailor your paper to follow the unspoken “rules” of that research area.
In these cases, it is invaluable to have access to a senior researcher (someone who is an expert in the domain) to help navigate that world. The issue, of course, is that such people are few and far between. The only way to have access to them is to join a reputable lab (either in industry or academia) as an intern/other junior role. This is an ugly fact about ML research, and raises the barrier of entry into the field.</p>
<h3 id="focus-on-reading-only-relevant-papers">Focus on reading only relevant papers</h3>
<p>Because of the volume of new ML papers, it is important to learn how to selectively read papers that are relevant to you (<a href="http://ccr.sigcomm.org/online/files/p83-keshavA.pdf">this article is a great tutorial</a>). When you do so, you’ll notice that most papers you come across are just not relevant to your work. Also, many newly-proposed techniques only work in specific settings that they were developed to solve. <strong><em>Really great papers can be identified by popularity (citation count, awards), novelty of the ideas, or consolidation of many ideas in the domain</em>.</strong></p>
<h3 id="you-dont-need-to-be-an-expert-to-contribute-to-the-state-of-the-art">You don’t need to be an expert to contribute to the State-of-the-art</h3>
<p>I was hesitant about writing papers on topics I was new to…what if I got an important concept wrong and made a fool of myself?</p>
<ul>
  <li>In this regard, I’ve found <a href="https://www.nature.com/articles/426389a.pdf">the advice of the late Steven Weinberg</a> very motivational. To paraphrase, <strong><em>the first step to doing good research, is doing okay research and asking an expert to review it</em></strong>. So don’t be afraid about putting out research to the best of your ability! Processes like peer-review are meant to give feedback on work which does not meet the bar.</li>
  <li>Additionally, <strong><em>your research ideas do not need to be complicated; they must be novel and your experiments must rigorously show that they work</em>.</strong> A lot of problems cannot be solved with simpler techniques, which is why things like the LSTM and Transformer exist. However, have you heard of K-fold cross-validation? It’s a simple, extremely popular idea which greatly improves the generalization performance of ML models. Focus on results, not complexity.</li>
</ul>

<h3 id="get-your-experiments-right">Get your experiments right</h3>
<p><strong><em>Test your ideas as fast as humanly possible</em></strong> <em>;</em> as a scientist, this is the main way to measure how productive you are on a day-to-day basis.
But <strong><em>be careful about what your experiments are actually measuring</em></strong>. “Experimental design” is a whole field, and unlike writing code, there is an unambiguously “right” and “wrong” way to run experiments. When in doubt, ask someone to review your experimental methodology <em>before</em> spending the time to generate results.</p>
<h3 id="writing-papers-teaches-you-the-tricks-of-the-trade">Writing papers teaches you the tricks of the trade</h3>
<p>Once you start writing papers, you will realize a few truths about scientific writing:</p>
<ul>
  <li>Papers are difficult to read because (a) they try to condense a huge amount of information into a limited space, which not everyone is good at; (b) authors try to sound professional even when discussing very simple ideas.</li>
  <li><strong><em>The paper you are currently reading is derived from the ideas of one or two cited papers…figure out which ones those are and read them if needed.</em></strong></li>
  <li>None of the authors have read <em>all</em> the research papers that they are citing in detail. A lot of citations are meant to establish background knowledge about the problem or “obvious” facts in the domain. If a large percentage of these facts are new information to you, then take a step back and learn the background first.</li>
</ul>

<h3 id="focus-on-how---and-where---you-present-your-work">Focus on how - and where - you present your work</h3>
<ul>
  <li><strong><em>The impact of your ideas is highly dependent on how well you communicate them, particularly in written form</em></strong>(papers, technical documents, etc). If your English skills are poor, work to improve them: “<a href="https://www.amazon.in/Elements-Style-William-Strunk-Jr/dp/020530902X">The Elements of Style</a>” by Strunk and White is a classic reference.</li>
  <li><strong><em>The selectivity of the venue where you publish is a yardstick for the quality of the work you are capable of producing.</em></strong> One paper in a Tier-1 conference is worth more than 3-4 papers at Tier-3 conferences.</li>
</ul>

<p>There’s a lot of things this article has not covered…as I mentioned, it’s a journey which contains a lot of “small” realizations about yourself and the field. However, I hope this provides some insight on the important part of this journey. Good luck!</p>]]></content><author><name>Abhishek Divekar</name></author><category term="career" /><category term="machine-learning" /><category term="research" /><summary type="html"><![CDATA[How I moved from software engineering to Applied Science at Amazon without a Masters or PhD: personal investment, ML skills, knowing the job, and learning to be a scientist.]]></summary></entry></feed>