Skip to content

Commit

Permalink
Merge 3.5 changes before release (#130)
Browse files Browse the repository at this point in the history
* Broadsea 3.0 merge into develop (#52)

* Initial commit of adding SSL to traefik and Solr

* Added Ares profile, using forked version

* Added profiles, support for git-based webapi and atlas builds

* Fixed WebAPI from git. Began security env variable sections. Added docker arch variable.

* Fixed yml error

* Added AD section

* Added more security provider env vars, some cleanup

* Added volume for cacerts (LDAPS)

* Removed original config-local

* Added security provider enabled entries, default values for sec items

* New README, removed extraneous db specific WebAPI stuff, organized .env into sections

* Fixed md issue in readme

* Added ldaps item to readme

* Clarification on ssl in readme

* Added OMOP vocab and phoebe loading services. With updated readme

* Workaround for if cacerts file isn't present

* Removed tmpfs assignment from postgres-load

* Fixes for phoebe unzip, clarity on vocab and phoebe vars

* Added vocab cpt conversion option based on if env var is fulfilled

* Added Ares data folder env and updated README

* Using new MAVEN_PARAMS in webapi from git. Renamed SOLR_ENDPOINT.

* Added dynamic splash page, ares entry, changed routes for content and hades. Added env var for certs folder.

* Updated hades and atlasdb to amd64 arch

* Added restart attribute to solr vocab

* Fixes #62, images were blocked due to content security policy meta tag

* Fixes div show/hide by removing extra quotes

* Updated Ares copy

* Tweak to hades config for backslash issue. Updated Mac Silicon language in readme

* Added .jar to gitignore

* Added openldap and posit connect services

* Adds OpenLDAP, Open Source Shiny Server, and Posit Connect services. Updates Traefik to new stable release.

* Added atlas cohort cache settings. Added mount option for jdbc jar files in webapi.

* Removed unnecessary flyway migration params. Updated comment on Atlas disable browser check

* Reverted flyway changes for now.

Added Broadsea content title variable.

Added WebAPI multi-language variable.

Added WebAPI security token expiration variable.

* Initial support of execution engine in webapi and atlas

* Added option for flyway baseline on migrate. Removed faulty localStorage clear operation in atlas config-local.

* Added initial achilles container

* Added achilles ares export

* Added logging level variables for webapi

Shifted jdbc drivers to a new docker volume path

Added security oid customparams setting for webapi

* Changed ares git location from fork to OHDSI repo

* Syncing with issue 96 fix.

Removed OID customparams variable for now.

* Fix omop load script to use UMLS key when available

Fix Solr Vocab import step to have command on 1 line

Added vocab pg files path env var

Brought back security oid customparams env var

Updated alpine to 3.18.2 for pg load containers to fix zip install issue

Added phoebe gitignores for expanded files

* Added more Atlas configLocal settings

* Added dbt, achilles, dqd, aresIndexer.

Moved other compose files into compose folder.

Changed broadsea content folder name.

Enabled traefik dashboard.

* Shifted to local Dockerfiles for post processing services

* Added perseus services.

Removed deprecated postgresql folder for webtools.

Added arachne golden file creation - still needs work.

* Added docker secrets to all services, except the tokens for perseus-user

Still needs testing.

* Shifted secrets toward specific file env vars

* Added github action for default profile

* Added missing dbt vars to env

* Added missing default value for phoebe vocab files in env

* Changed patterns for bringing jdbc files for solr and webapi

Added none.jar placeholder to jdbc folder

* Relative path issue for webapi none.jar

* Updated README file

Changed env file to use ares shared volume by default for ares web

* Introducing quarto for content page

* content typos

For now removing the subdirectory from default ares data path

* added wget install back

* apt-get issue

* Using RUN over EOF syntax

* Adjustments to tests

* checkout action with branch

Added Perseus testing

* Fixed image issue with posit connect, errant path issue in webapi jdbc file path

* trying classpath-based method to additional webapi jar

* [skip ci] copy update in env

* removed ref tag from ci files

* Default tests split up, perseus test improved

Fixes for webapi additional jdbc

* Added debug to webapi container

* Fix to atlasdb env

* Using fork of atlasdb

* added branch name to atlasdb fork

* Using new atlasdb image with secret

Removed option to set atlasdb password

* Adding sleep hack for atlasdb in github action

* Adding more sleep and psql test

* Disabled mac

* added sudo

* Adding public webapi test

* sleep on webapi build

* Adding path to env

* Adding relative env path

* ip address echo and certs folder create

* with env_file set

* Attempting separate config files

* Changes to traefik dynamic yml, separate tls out

* Reordering steps, curl fails fast

* Trying public ip for broadsea host

* revert atlasdb test to localhost

* debug webapi with logs

* Trying non-secret atlasdb

* more pw debug

* added sleep back

* readme as debug

* trying to write file

* testing only macos

* testing atlasdb password

* shifting back to linux

* changed test to use secret file

* testing without password set

* adding webapi back, but trying to ensure env is used correctly

* fix of cd command

* Fixes to secrets in webapi and solr

* default solr vocab to disabled.

Added curl webapi test back.

* added mac psql.

trying simple default test.

* Adding sleep back for WebAPI test.

Fix to atlas curl url.

* Trying https in test

* fix for self signed action

* trying GA with default ssl params

* changing ssl test to linux only

* fix for runner os

* removed macos

* trying new openssl steps

* copying certs before use

* trying sudo

* hostkey fix

* trying to add crt to ca

* using manual ssl creation

* added ca step back

* adding insecure flag

* Adding mac back

* Trying more jobs

* Added needs

* combined into 1 workflow

changed ldap to disabled by default

* moved openldap and solr to separate test ymls

* Fixes for perseus web - to override port

Added atlasdb to solr vocab testing

* Reverted to standard perseus web and backend

updated port of backend

* Adds new cdm-builder settings file

* Initial achilles GA test

Removed extra sleep from openldap GA test.

* Added achilles log test to GA

Renamed some GA files

Added db auth to webapi GA test

Updated secret for db auth pw

* Added sleep to achilles GA, various renaming of GAs

* Reorg of atlas files, updated quarto content. Preparing for add cdm profile.

Removed arachne for now.

* comment out new cdm password secrets

* Added improved cacerts method.
Added Snowflake PK support.
Added Github PAT support.

* Change from < to cat in omop vocab script

* Updated < to cat in phoebe script

* Added dummy github pat secret file

* Adding apk update to load omop script

* Updated readme to add BYO JDBC driver

* Update Basic Authentication  (#113)

* changing default password to match the broadsea-atlasdb POSTGRES_PASSWORD

* updating .env values for basic security

* adding an authentication query env var

* adding setup instructions for basic security

---------

Co-authored-by: Ajit Londhe <londhe@ohdsi.org>

* Cleanup of readme, some new notes

* Adding GITHUB TOKEN to Achilles GA script

* Trying to use org secret for Github pat

* Trying to set github pat as env within achilles dockerfile

* shift achilles github pat secret to build

* Added secrets mount in achilles dockerfile

* Trying docker compose (no hyphen) in achilles ci

* Adopt Docker compose v2 and hyphen-less docker compose

* Removal of extraneous ports, removal of perseus-web, disable traefik dashboard

Add MAVEN_PROFILE to webapi-from-git pattern

* Re-enabled traefik dashboard. Added copy to Readme about Perseus

* Simplify perseus ci test to just http

* README editing (#121)

* cmd editing and highlighting for shell

* added codeblock highlighting

* formatting

* transformed profiles nested lists to tables

* Added contents section for navigation

* expanded on Shutdown section

* Fixed cacerts item in Readme. Restore certs folder step in perseus ci.

* Adding missing config-gis.js file to Atlas container

* Added jupyter notebook profile. Fixes for Achilles Dockerfile and traefik routing.

Update traefik to 2.11.2.

Set atlasdb to restart unless stopped.

* Added pgadmin from @leeevans commits and updated the content page to show it

Also borrowed some WebAPI DB security improvements from @leeevans

* Updating to 3.5 release number

* Changing OSs in CI matrices

* Using only Linux OS for CI for now. Mac image runners not functioning properly.

---------

Co-authored-by: Will Roddy <45157996+wtroddy@users.noreply.github.com>
Co-authored-by: Hayden B Spence <76969160+haydenbspence@users.noreply.github.com>
  • Loading branch information
3 people authored Apr 29, 2024
1 parent 780f3a6 commit 1418191
Show file tree
Hide file tree
Showing 102 changed files with 3,562 additions and 534 deletions.
275 changes: 235 additions & 40 deletions .env

Large diffs are not rendered by default.

27 changes: 27 additions & 0 deletions .github/workflows/achilles.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
name: achilles profile (Linux)
on: push
jobs:
achilles:
strategy:
matrix:
os: [ubuntu-22.04]
httpType: [http]
runs-on: ${{ matrix.os }}
steps:
- name: Setup Docker on macOS
if: runner.os == 'macOS'
uses: douglascamata/setup-docker-macos-action@v1-alpha
- name: Checkout
uses: actions/checkout@v3
- name: Create certs folder
run: mkdir -p ./certs
- name: Write Github PAT secret file
run: echo ${{ secrets.GH_TOKEN }} > ./secrets/github_pat
- name: Build and run achilles container
run: docker compose --env-file .env --profile achilles up -d
- name: Use busybox container to grab log file and verify no errors
run: |
sleep 30
CID=$(docker run -d -v cdm-postprocessing-data:/cdm-postprocessing-data busybox true)
docker cp $CID:/cdm-postprocessing-data/achilles/data/demo_cdm/log_achilles.txt ./log_achilles.txt
if grep -q "ERROR" ./log_achilles.txt;then exit 1;else exit 0;fi
54 changes: 54 additions & 0 deletions .github/workflows/default.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
name: default profile (Linux)
on: push
jobs:
default:
strategy:
matrix:
os: [ubuntu-22.04]
httpType: [http, https]
runs-on: ${{ matrix.os }}
steps:
- name: Setup Docker on macOS
if: runner.os == 'macOS'
uses: douglascamata/setup-docker-macos-action@v1-alpha
- name: Checkout
uses: actions/checkout@v3
- name: Create certs folder
run: mkdir -p ./certs
- name: Replace HTTP_TYPE if https (Linux)
if: matrix.httpType == 'https' && runner.os == 'Linux'
run: sed -i 's/HTTP_TYPE=\"http\"/HTTP_TYPE=\"https\"/g' .env
- name: Replace HTTP_TYPE if https (Mac)
if: matrix.httpType == 'https' && runner.os == 'macOS'
run: sed -i '' 's/HTTP_TYPE=\"http\"/HTTP_TYPE=\"https\"/g' .env
- name: Create and trust self signed certs
if: matrix.httpType == 'https'
run: |
openssl req -x509 -out ./certs/broadsea.crt -keyout ./certs/broadsea.key \
-newkey rsa:2048 -nodes -sha256 \
-subj '/CN=127.0.0.1' -extensions EXT -config <( \
printf "[dn]\nCN=127.0.0.1\n[req]\ndistinguished_name = dn\n[EXT]\nsubjectAltName=DNS:127.0.0.1\nkeyUsage=digitalSignature\nextendedKeyUsage=serverAuth")
- name: Build traefik container
run: docker compose --env-file .env up -d
- name: Build default profile
run: docker compose --env-file .env --profile default up -d
- name: Setup postgres client on Linux
if: runner.os == 'Linux'
run: sudo apt-get install --yes --no-install-recommends postgresql-client
- name: Setup postgres client on MacOS
if: runner.os == 'macOS'
run: brew install libpq
- name: Test atlasdb
run: PGPASSWORD=$(cat ./secrets/webapi/WEBAPI_DATASOURCE_PASSWORD) psql -h 127.0.0.1 -p 5432 -U postgres -c "select source_name from webapi.source limit 1"
- name: Test traefik container
run: docker logs traefik
- name: Test content container
run: curl -s -k --retry 10 --retry-connrefused ${{matrix.httpType}}://127.0.0.1
- name: Test HADES
run: curl -s -k --retry 10 --retry-connrefused ${{matrix.httpType}}://127.0.0.1/hades/auth-sign-in
- name: Test Atlas
run: curl -s -k --retry 10 --retry-connrefused ${{matrix.httpType}}://127.0.0.1/atlas/#/home
- name: Test WebAPI info endpoint
run: |
sleep 30
curl -s -k --retry 10 --retry-connrefused ${{matrix.httpType}}://127.0.0.1/WebAPI/info
27 changes: 27 additions & 0 deletions .github/workflows/perseus.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
name: perseus profile (Linux)
on: push
jobs:
perseus:
strategy:
matrix:
os: [ubuntu-22.04]
httpType: [http]
runs-on: ${{ matrix.os }}
steps:
- name: Setup Docker on macOS
if: runner.os == 'macOS'
uses: douglascamata/setup-docker-macos-action@v1-alpha
- name: Checkout
uses: actions/checkout@v3
- name: Create certs folder
run: mkdir -p ./certs
- name: Build traefik container
run: docker compose up -d
- name: Check traefik container
run: docker logs traefik
- name: Build perseus profile
run: docker compose --profile perseus up -d
- name: Test perseus frontend
run: |
sleep 30
curl -s --retry 10 --retry-connrefused http://127.0.0.1/perseus
44 changes: 44 additions & 0 deletions .github/workflows/solrvocab.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
name: solr-vocab-no-import profile (Linux)
on: push
jobs:
solrvocab:
strategy:
matrix:
os: [ubuntu-22.04]
httpType: [http, https]
runs-on: ${{ matrix.os }}
steps:
- name: Setup Docker on macOS
if: runner.os == 'macOS'
uses: douglascamata/setup-docker-macos-action@v1-alpha
- name: Checkout
uses: actions/checkout@v3
- name: Create certs folder
run: mkdir -p ./certs
- name: Replace HTTP_TYPE if https (Linux)
if: matrix.httpType == 'https' && runner.os == 'Linux'
run: sed -i 's/HTTP_TYPE=\"http\"/HTTP_TYPE=\"https\"/g' .env
- name: Replace HTTP_TYPE if https (Mac)
if: matrix.httpType == 'https' && runner.os == 'macOS'
run: sed -i '' 's/HTTP_TYPE=\"http\"/HTTP_TYPE=\"https\"/g' .env
- name: Create and trust self signed certs
if: matrix.httpType == 'https'
run: |
openssl req -x509 -out ./certs/broadsea.crt -keyout ./certs/broadsea.key \
-newkey rsa:2048 -nodes -sha256 \
-subj '/CN=127.0.0.1' -extensions EXT -config <( \
printf "[dn]\nCN=127.0.0.1\n[req]\ndistinguished_name = dn\n[EXT]\nsubjectAltName=DNS:127.0.0.1\nkeyUsage=digitalSignature\nextendedKeyUsage=serverAuth")
- name: Build traefik container
run: docker compose --env-file .env up -d
- name: Build atlasdb profile
run: docker compose --env-file .env --profile atlasdb up -d
- name: Build empty SOLR instance
run: docker compose --env-file .env --profile solr-vocab-no-import up -d
- name: Build webapi profile with SOLR_VOCAB enabled
run: SOLR_VOCAB_ENDPOINT=http://broadsea-solr-vocab:8983/solr docker compose --env-file .env --profile webapi-from-image up -d
- name: Test WebAPI info endpoint
run: |
sleep 30
curl -s -k --retry 10 --retry-connrefused ${{matrix.httpType}}://127.0.0.1/WebAPI/info
- name: Test SOLR endpoint
run: curl -s -k --retry 10 --retry-connrefused ${{matrix.httpType}}://127.0.0.1/solr/
59 changes: 59 additions & 0 deletions .github/workflows/webapi_auth.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
name: WebAPI Authentication (Linux)
on: push
jobs:
openldap:
strategy:
matrix:
os: [ubuntu-22.04]
httpType: [http]
runs-on: ${{ matrix.os }}
steps:
- name: Setup Docker on macOS
if: runner.os == 'macOS'
uses: douglascamata/setup-docker-macos-action@v1-alpha
- name: Checkout
uses: actions/checkout@v3
- name: Create certs folder
run: mkdir -p ./certs
- name: Build traefik container
run: docker compose --env-file .env up -d
- name: Build atlasdb profile
run: docker compose --env-file .env --profile atlasdb up -d
- name: Build webapi profile with ldap enabled
run: SECURITY_AUTH_LDAP_ENABLED=true docker compose --env-file .env --profile webapi-from-image up -d
- name: Test WebAPI info endpoint
run: |
sleep 30
curl -s -k --retry 10 --retry-connrefused ${{matrix.httpType}}://127.0.0.1/WebAPI/info
- name: Build OpenLDAP
run: docker compose --env-file .env --profile openldap up -d
- name: Test openldap auth in WebAPI
run: |
curl -d "login=user1&password=$(<./secrets/openldap/OPENLDAP_ACCOUNT_PASSWORDS)" -H "Content-Type: application/x-www-form-urlencoded" -X POST ${{matrix.httpType}}://127.0.0.1/WebAPI/user/login/ldap
db:
strategy:
matrix:
os: [ubuntu-22.04]
httpType: [http]
runs-on: ${{ matrix.os }}
steps:
- name: Setup Docker on macOS
if: runner.os == 'macOS'
uses: douglascamata/setup-docker-macos-action@v1-alpha
- name: Checkout
uses: actions/checkout@v3
- name: Create certs folder
run: mkdir -p ./certs
- name: Build traefik container
run: docker compose --env-file .env up -d
- name: Build atlasdb profile
run: docker compose --env-file .env --profile atlasdb up -d
- name: Build webapi profile with ldap enabled
run: SECURITY_AUTH_JDBC_ENABLED=true docker compose --env-file .env --profile webapi-from-image up -d
- name: Test WebAPI info endpoint
run: |
sleep 30
curl -s -k --retry 10 --retry-connrefused ${{matrix.httpType}}://127.0.0.1/WebAPI/info
- name: Test db auth in WebAPI
run: |
curl -d "login=ohdsi&password=$(<./secrets/webapi/SECURITY_DB_DATASOURCE_PASSWORD)" -H "Content-Type: application/x-www-form-urlencoded" -X POST ${{matrix.httpType}}://127.0.0.1/WebAPI/user/login/db
7 changes: 6 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
.DS_Store
omop_vocab/files
.env_local
*.jar
*.lic
phoebe/concept_recommended.csv
phoebe/__MACOSX
atlas/add_cdm
.Rproj.user
strategus/
Loading

0 comments on commit 1418191

Please sign in to comment.