<p></p><h2>Background</h2><p>The blog explains how to create a scaled analytics environment for both python jupyter notebook and neo4j. This would be a solution In case you are in a following situation</p><ul><li>You won't be able to use typical cloud platform because of a company data policy, but you would need to prepare scaling platform by yourselves</li><li>You would like to create microservices connecting each container with more customized way</li></ul><p><br></p><h2>What is JupyterHub?</h2><p>Mai Ngoc summarized well in <a href="https://kienmn97.medium.com/manually-deploy-jupyterhub-on-kubernetes-for-a-single-machine-dbcd9c9e50a4" target="_blank">a medium post in Apr 2020</a>.</p><blockquote><p>JupyterHub brings the power of notebooks to groups of users. It is a set of processes that together provide a single user Jupyter Notebook server for each person in a group. There are three major subsystems in JupyterHub:<br><br>1. Hub : manages user accounts, authentication, and coordinates Single User Notebook Servers using a Spawner<br>2. Proxy : the public facing part of JupyterHub that uses a dynamic proxy to route HTTP requests to the Hub and Single User Notebook Servers. Configurable http proxy is the default proxy<br>3. Single User Notebook Server : a dedicated, single user, Juptyer Notebook server is started for each user on teh system when the user logs in. The object that starts the single user notebook servers is called a spawner</p></blockquote><p><img src="/media/django-summernote/2022-04-17/8433c69a-0bed-4f36-a882-81638ac4a4bb.png" style="width: 636px;"><br></p><p><span style="font-size: 10px;">picture from :&nbsp;https://jupyterhub.readthedocs.io/en/stable/reference/technical-overview.html</span><span style="font-size: 11px;"></span></p><p><span style="font-size: 14px;"></span></p><p>The basic operation of jupyterhub are as you see above;</p><ol><li>The Hub spawns the proxy</li><li>The proxy forwards all requests to hte Hub by default</li><li>The Hub handles login, and spwns single-user notebook servers on demand</li><li>The Hub configures the proxy to forward url predixes to single-user notebook servers</li></ol><p><br></p><h2>Prerequisite</h2><p>We will use JupyterHub helm to deploy JupyterHub to local kubernetes Minikube. You need to install;</p><ol><li>Minikube</li><li>Kubectl</li><li>Helm</li></ol><p><br></p><h2>Deployment Procedure</h2><p>1. Start minikube</p><pre><p># shell minikube start</p></pre><p>2. Create project directly</p><pre># shell mkdir jhub &amp;&amp; cd jhub<br></pre><p>3. Install JupyterHub helm</p><pre># shell helm repo add jupyterhub <a href="https://jupyterhub.github.io/helm-chart/" target="_blank">https://jupyterhub.github.io/helm-chart/</a> helm repo update</pre><p>This would show output in shell like:</p><pre># shell Hang tight while we grab the latest from your chart repositories... ...Skip local chart repository ...Successfully got an update from the "stable" chart repository ...Successfully got an update from the "jupyterhub" chart repository Update Complete. ⎈ Happy Helming!⎈<br></pre><p>You can check installed helm chart by;</p><pre># shell helm repo list</pre><p>4. Check name space</p><pre># shell kubectl get namespace</pre><p>Create namespace if you woud like</p><pre>kubectl create namespace &lt;your new namespace&gt;</pre><p>Now set whatever namespace as the default context</p><pre>kubectl config set-context --current --namespace=&lt;type specific namespace from the list&gt;</pre><p>5. Create config file. You don't have to put anything inside at this time. Make sure you are under project directry you've created in step 2</p><pre># shell touch config.yaml</pre><p>6. Deploy jupyterhub from helm and config.yaml</p><pre># shell helm install &lt;RELEASE_NAME&gt; jupyterhub/jupyterhub -f config.yaml </pre><p>You can put anything in &lt;RELEASE_NAME&gt;(for example, `jhub`) but you have to remember.</p><p><br></p><p>Wait for deploying this, and you will see hub and proxy pods in a certain namespace cluster</p><pre># shell kubectl get pod</pre><p>you will see something like;</p><pre>NAME READY STATUS RESTARTS AGE hub-58985485c7-qsknh 1/1 Running 1 38h proxy-68df9f866f-xxgpb 1/1 Running 9 38h<br></pre><p>7. Get external IP address to access</p><pre># shell kubectl get service</pre><p>you will get something like to get a port:</p><pre>NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE hub ClusterIP 10.102.82.41 &lt;none&gt; 8081/TCP 38h proxy-api ClusterIP 10.98.42.187 &lt;none&gt; 8001/TCP 38h proxy-public LoadBalancer 10.99.81.177 &lt;pending&gt; 80:30361/TCP 38h<br></pre><p>Get minikube external IP that we could access outside of cluster</p><pre># shell minikube ip</pre><p>Put &lt;minikube ip&gt;:port to browser to access. My case is `http://192.168.64.2:30361`. Then you should get below login window</p><p><img src="/media/django-summernote/2022-04-18/982b50bf-f027-4c4a-bb49-b06850afbec6.png" style="width: 1098px;"><br></p><p><br></p><p><span style="color: inherit; font-family: inherit; font-size: 30px;">Login to Hub and Connect to Neo4j Container</span><br></p><p>1. At login window, create your username and password. This will create a new pod for the user</p><pre>NAME READY STATUS RESTARTS AGE jupyter-tak 1/1 Running 1 25h<br></pre><p>Assuming you have neo4j container in a same cluster, you could connect thru by;</p><p><img src="/media/django-summernote/2022-04-19/a5a154af-42a9-434f-b699-11dba7f8526c.png" style="width: 1095px;"><br></p><p>Make sure that you put <b>cluster IP </b>from neo4j services. Confirmed that tak-neo or tak-neo-neo4j's cluster IP work</p><pre>NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE tak-neo ClusterIP <b>10.101.133.16</b> &lt;none&gt; 7687/TCP,7474/TCP,7473/TCP 7m57s tak-neo-admin ClusterIP 10.108.4.202 &lt;none&gt; 6362/TCP,7687/TCP,7474/TCP,7473/TCP 7m57s tak-neo-neo4j LoadBalancer <b>10.104.171.197</b> &lt;pending&gt; 7474:32008/TCP,7473:31040/TCP,7687:32557/TCP 7m57s<br></pre><p>and you will see the new database at neo4j browser as well</p><p><img src="/media/django-summernote/2022-04-18/84d38cd2-d19b-43af-9687-dd312c9685d8.png" style="width: 1258px;"><br></p><p><br></p><h2>Customization and Miscellaneous</h2><p>All customization will be written in future blog but below are quick references</p><ol><li>Additional dependencies :&nbsp;<a href="https://zero-to-jupyterhub.readthedocs.io/en/latest/jupyterhub/customizing/user-environment.html#customize-an-existing-docker-image" target="_blank">https://zero-to-jupyterhub.readthedocs.io/en/latest/jupyterhub/customizing/user-environment.html#customize-an-existing-docker-image</a></li><li>Default UI to jupyter lab :&nbsp;<a href="https://zero-to-jupyterhub.readthedocs.io/en/latest/jupyterhub/customizing/user-environment.html#use-jupyterlab-by-default" target="_blank">https://zero-to-jupyterhub.readthedocs.io/en/latest/jupyterhub/customizing/user-environment.html#use-jupyterlab-by-default</a></li><li>How to create persistent storage in JupyterHub :&nbsp;<a href="https://kienmn97.medium.com/persistent-storage-in-jupyterhub-on-kubernetes-cluster-running-on-minikube-4b469bdb1b86" target="_blank">https://kienmn97.medium.com/persistent-storage-in-jupyterhub-on-kubernetes-cluster-running-on-minikube-4b469bdb1b86</a></li><li>Add spark :&nbsp;</li></ol><p><br></p><p>How to uninstall deployment</p><pre># shell helm uninstall &lt;RELEASE_NAME&gt;</pre>If you don't remember &lt;RELEASE_NAME&gt;,<p></p><p></p><pre># shell helm list</pre><p>then you will see existing deployment by helm like:</p><pre>NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION tak-jhub default 1 2022-04-16 23:30:35.983417 -0700 MST deployed jupyterhub-1.2.0 1.5.0<br></pre><p><br></p><p>how to upgrade helm chart after updating configuration</p><pre># shell helm upgrade &lt;RELEASE_NAME&gt; jupyterhub/jupyterhub -f config.yaml</pre><p><br></p><p>This blog is mainly referencing&nbsp;<a href="https://zero-to-jupyterhub.readthedocs.io/en/latest/index.html" target="_blank">zero-to-jupyterhub</a>, and <a href="https://kienmn97.medium.com/manually-deploy-jupyterhub-on-kubernetes-for-a-single-machine-dbcd9c9e50a4" target="_blank">the medium post</a></p><p></p>