feat: Add keycloak (#510)

* feat: Add keycloak

* fix lint error

* Deploy keycloak in default

* Fix role names in deps

* Remove dynamic key in ansible variable dic

* Use custom wait logic because postgresql CRD doesn't have condition status

* Use production mode for keycloak

* Wait until zalando operator ready

* Set admin password explicitly

* Fix ingress config

* Create grafana client in keycloak and enable oauth in grafana

* Use PXC instead of zalando postgres

* Set default value for atmosphere_keycloak_enabled

* Split the mysql queries into multiple parts

* Solve convo

* Use ansible module for keycloak realm

* Concatenate realm arrays

* Tune keycloak config to work with mysql vendor and fix keycloak db user's host config

* Add notes about mysql vendor support in keycloak

* remove default client scope setting

it requires community.general collection upgrade to 4.7.0.
But there are other collections in the deps list which use lower versions.

Need to bump its version at some time surely.

* Tuen grafana oauth config

* Fix keycloak client auth mode

* Use a variable for keycloak database name

* Fix lint error

* Remove unused var from doc

* Manage pxc strict mode in ansible

* revert the changes out of the scope

* ignore changes during realm creation

* Resolve commnents

* Fix default values

* Use official keycloak image instead of bitnami

* Set proxy mode using KC env var

* securely reference oauth secrets in grafana.ini

* Remove implicit octal value in helm values

* Support id token in keycloak clients for rbac and add ingress role

* Update grafana role map attribute

* Use j2 template

https://stackoverflow.com/questions/63961938/ansible-variable-conversion-to-int-is-ignored

* Create client roles

* Fix yaml lint error

* Add keycloak health check in consumer roles

* fix: use correct annotations

* chore: add keystone-keycloak-backend to keystone

* fix: make openstack_helm_endpoints work cleaner

* chore: clean up unecessary docs

* fix: lock down grafana to allow users with roles only

* feat: integrate keystone with keycloak

* chore: added horizon auth via keycloak

* chore: add slo for grafana

* chore: bump to 0.1.5

* Fix yaml lint error

* Fix client role creation

* chore(kube-prometheus-stack): update to latest

* fix: KubeJobFailed should be SEV-3

* chore: refactor softnet alerts

* chore(monitoring): migrate to using jsonnet

* chore: refactor alerts

* chore: major monitoring refactor

* fix: solve alerts

* fix: apiserver selector

* more cleanups

* switch from SEV- to P

* fix: improve port binding alerts

* fix admin state alert for neutron

* map some more alerts

* drop uuid

* Revert "drop uuid"

This reverts commit ad0f05d0e7564759e8259c2cc53c2e2f5c73e1b8.

* fix: drop recording rules

* switch alertmanager to jsonnet

* fix: idempotence for monitoring

* chore: fix linters

* chore: lower ceph osd timeouts to p4

* chore: refactor to using new jsonnet

* chore: use vendor path

* chore: fix mixin for alertmanager

* core: fix selector

* ci: add initial keycloak kind tests

* ci: run keycloak basic scenario

* chore: refactor to using multiple domains

* chore: wip for keycloak multidomain

* adding multi domain support for keycloak as external identity provider (#556)

Co-authored-by: Mohammed Naser <mnaser@vexxhost.com>

* chore: initial impl for multiple domain

* chore: grafan + keycloak wip

* chore: fix missing roles

* chore: remove commented out role

* chore: add ci debug

* chore: use smaller nodes for test

* ci: fix keycloak_user_info

* ci: fix keycloak_user_info

* ci: debug keycloak ci

* chore: fix linters

* ci: retry a few times for keycloak users to appear

* ci: ci fixes

* ci: misc fixes

* ci: fix secret generation

* ci: fix missing secrets

---------

Co-authored-by: okozachenko1203 <okozachenko1203@users.noreply.github.com>
Co-authored-by: Mohammed Naser <mnaser@vexxhost.com>
Co-authored-by: Jeremy Lee <6729613+legit-ninja@users.noreply.github.com>
181 files changed
tree: 83c8a77e08e5f42aa96b484e7caf5c51c958f004
  1. .github/
  2. build/
  3. charts/
  4. cmd/
  5. docs/
  6. hack/
  7. images/
  8. internal/
  9. meta/
  10. molecule/
  11. playbooks/
  12. plugins/
  13. roles/
  14. tests/
  15. .ansible-lint
  16. .coveragerc
  17. .flake8
  18. .gitignore
  19. .markdownlint.yaml
  20. .pre-commit-config.yaml
  21. .python-version
  22. .release-please-manifest.json
  23. CHANGELOG.md
  24. Earthfile
  25. galaxy.yml
  26. go.mod
  27. go.sum
  28. mkdocs.yml
  29. poetry.lock
  30. pyproject.toml
  31. README.md
  32. release-please-config.json
README.md

Atmosphere

Community

If you have any questions and discussions about Atmosphere, you can join the community:

Quick Start

The quick start intends to provide the most near-production experience possible, as it is architected purely towards production-only environments. In order to get a quick production-ready experience of Atmosphere, you will need access to an OpenStack cloud.

The quick start is powered by Molecule and it is used in continuous integration running against the VEXXHOST public cloud so that would be an easy target to use to try it out.

You will need the following quotas set up in your cloud account:

  • 8 instances
  • 32 cores
  • 128GB RAM
  • 360GB storage

These resources will be used to create a total of 8 instances broken up as follows:

  • 3 Controller nodes
  • 3 Ceph OSD nodes
  • 2 Compute nodes

First of all, you'll have to make sure you clone the repository locally to your system with git by running the following command:

git clone https://github.com/vexxhost/atmosphere

You will need poetry installed on your operating system. You will need to make sure that you have the appropriate OpenStack environment variables set (such as OS_CLOUD or OS_AUTH_URL, etc.). You can also use the following environment variables to tweak the behaviour of the Heat stack that is created:

  • ATMOSPHERE_STACK_NAME: The name of the Heat stack to be created (defaults to atmosphere).

  • ATMOSPHERE_PUBLIC_NETWORK: The name of the public network to attach floating IPs from (defaults to public).

  • ATMOSPHERE_IMAGE: The name or UUID of the image to be used for deploying the instances (defaults to Ubuntu 20.04.3 LTS (x86_64) [2021-10-04]).

  • ATMOSPHERE_INSTANCE_TYPE(Deprecated): The instance type used to deploy all of the different instances.(It doesn't have its own default value.) This has been deprecated from v1.4.0. You can configure the instance type per a machine role using ATMOSPHERE_CONTROLLER_INSTANCE_TYPE, ATMOSPHERE_COMPUTE_INSTANCE_TYPE, and ATMOSPHERE_STORAGE_INSTANCE_TYPE variables. For backwards compatibility, if variables specific to the machine roles are not set and ATMOSPHERE_INSTANCE_TYPE is set, ATMOSPHERE_INSTANCE_TYPE value is used.

  • ATMOSPHERE_CONTROLLER_INSTANCE_TYPE: The instance type used to deploy controller instances (defaults to v3-standard-16).

  • ATMOSPHERE_COMPUTE_INSTANCE_TYPE: The instance type used to deploy compute instances (defaults to v3-standard-4).

  • ATMOSPHERE_STORAGE_INSTANCE_TYPE: The instance type used to deploy storage instances (defaults to v3-standard-4).

  • ATMOSPHERE_NAMESERVERS: A comma-separated list of nameservers to be used for the instances (defaults to 1.1.1.1).

  • ATMOSPHERE_USERNAME: The username what is used to login into the instances ( defaults to ubuntu).

  • ATMOSPHERE_DNS_SUFFIX_NAME: The DNS domainname that is used for the API and Horizon. (defaults to nip.io).

  • ATMOSPHERE_ACME_SERVER: The ACME server, currenly this is from LetsEncrypt, with StepCA from SmallStep it is possible to run a internal ACME server. The CA of that ACME server should be present in the instance image.

  • ATMOSPHERE_ANSIBLE_VARS_PATH: The path for ansible group_vars and host_vars. This to build a multinode development cluster with own configs, that are not generated by molecule. This way you can test your configs before you bring them to production.

Once you're ready to get started, you can run the following command to install poetry dependencies:

poetry install

Then you can run the following command to build the Heat stack :

poetry run molecule converge

This will create a Heat stack with the name atmosphere and start deploying the cloud. Once it's complete, you can login to any of the systems by using the login sub-command. For exampel, to login to the first controller node, you can run the following:

poetry run molecule login -h ctl1

In all the controllers, you will find an openrc file location inside the root account home directory, as well as the OpenStack client installed there as well. You can use it by running the following after logging in:

source /root/openrc
openstack server list

The Kubernetes administrator configuration will also be available on all of the control plane nodes, you can simply use it by running kubectl commands on any of the controllers as root:

kubectl get nodes -owide

Once you're done with your environment and you need to tear it down, you can use the destroy sub-command:

poetry run molecule destroy

For more information about the different commands used by Molecule, you can refer to the Molecule documentation.

Contributing

You'll need to make sure that you have pre-commit setup and installed in your environment by running these commands:

pre-commit install --hook-type commit-msg