Goal#

The goal here is to setup a local development dockerized python project that utilises CI/CD to my linode VPS.

First Step#

First step is to get a simple local app running in a docker container. I’ve chosen the streamlit example app. (This is an app that I’ve previously dockerized and tested on the server. I want at this point to minimise the potential problems.)

First Problem#

The first problem I ran into was that github wouldn’t recognise my push requests. It seems github had changed their ssh key. More information here https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/

Once I got that sorted, I was ready to move forward.

Test for local development#

However, my streamlit example was pulling directly from the streamlit example git repo. So, I cloned locally, and updated the Dockerfile to use a local file rather than pull from a git repo.

This did bring up some interesting use cases for the future. #TODO

Faster Build#

I used a split up COPY command to allow my docker build to cache the pip install requirements and then copy the streamlit-example.py file if changed.

This greatly sped up the build time.

My finished Dockerfile was:

# Dockerfile

FROM python:3.9-slim

WORKDIR /app

RUN apt-get update && apt-get install -y \
    build-essential \
    curl \
    software-properties-common \
    git \
    && rm -rf /var/lib/apt/lists/*

COPY /app/requirements.txt .

RUN pip3 install -r requirements.txt

COPY /app .
# EXPOSE 8501

# HEALTHCHECK CMD curl --fail http://localhost:8080/_stcore/health

ENTRYPOINT ["streamlit", "run", "streamlit_app.py"]

And my docker-compose.yaml was:


version: '3.9'
services:
  streamlit_example:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: streamlit_example
    ports:
      - '8501:8501'
    networks:
      - nginx_default_network

networks:
  nginx_default_network:
    external: true

Here the network is set to the pre-established nginx docker network on the target VPS. (I’m using portainer to manage my docker dockererised sites on the VPS)

I ran the docker image locally.

It built.

I made some slight changes to the streamlit app file to test and rebuilt the docker image:

docker compose up --build

The changes I made to streamlit_app.py were applied and the image did not need to reinstall the requirements as stated in the requirements.txt.

Success so far.

Makefile#

I then wrote a Makefile.

If you get a *** missing separator. Stop. Error then maybe you’re using spaces to indent. Makefile requires tab to indent. Details here https://stackoverflow.com/questions/920413/make-error-missing-separator

# Makefile

build:
    docker compose up --build -d --remove-orphans
up:
    docker compose up -d
down:
    docker compose down
show_logs:
    docker compose logs

I ran the command:

make build

The docker image built, updated with changes to the streamlit_app.py file.

Nice.

I then pushed it to github and manually cloned it to my VPS into a new folder named “streamlit”.

I needed to sudo apt install make to run the Makefile, but it built.

I then changed the repository to private on github to test if it would still perform as before. It did.

Github Actions#

Back in my local project directory, I ran the following commands to get the CI/CD workflow setup. See the reference below for a full blog post on this written by Yash Prakash.

mkdir -p .github/workflows
touch .github/workflows/main.yml
name: Streamlit Example CI-CD

on:
  # Triggers the workflow on push or pull request events but only for the main branch

  push:
    branches: [main]

  pull_request:
    branches: [main]

  # Run this workflow manually from the Actions tab on Repo homepage
  workflow_dispatch:

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
    - name: Deploy to VPS
      uses: appleboy/ssh-action@master
      with:
        # VPS IP
        host: ${{ secrets.VPS_SSH_HOST }}

        # VPS username
        username: ${{ secrets.VPS_SSH_USERNAME }}

        # SSH key (copy it from your local machine)
        key: ${{ secrets.VPS_SSH_SECRET }}

	# SSH port
        port: ${{ secrets.VPS_SSH_PORT }}

	# passphrase
        passphrase: ${{ secrets.VPS_SSH_PASSPHRASE }}

	script: |
	  cd ${{ secrets.PROJECT_PATH }}
	  git pull origin main
	  make down
	  make build

This lead me to setting up a limited scope ssh user on my VPS to allow this specific github repo to access the hosting server.

I put together some notes on this here.

Next Step#

The next step will now be to develop a more in-depth CI/CD to function with a complex project (Dockerized Django - ninja-django - API) with a full suite of tests.

note: On April 21st, I got my code portfolio public git submodule to deploy and build using the same setup. It’s still a simple static site but it was a good second test of this process.

Helpful References:#

https://towardsdatascience.com/the-easy-python-ci-cd-pipeline-using-docker-compose-and-github-actions-80498f47b341