Container Registry

Options

With OpenShift, you can use a Container Registry which is internal or external to the cluster. What are the pros and cons of each option?

Internal Registry: (This is lives in the OpenShift Cluster, essentially like a local cache)
- Pros:
  1. No action is needed, as this out of the box for Cloud Pak for Data. All repo files and behaviors are set up for internal registry. Just add API Key and go.
  2. All images are local to the cluster, essentially like a cache of the IBM Repo.
- Cons:
  1. If you have more than one cluster, you will have multiple copies of the same images. While disk is cheap, a terabyte here and a terabyte there can add up over time.
  2. Upgrades you will pull images for every cluster vs once.
  3. There is limited controllability on which images you keep. You will retain every image even after you upgrade.
External Registry:
- Pros:
  1. One single repository for all of your clusters.
  2. Shorter time to install assemblies.
  3. Ability to stage upgrade testing using multiple repositories for testing.
  4. Periodic preload of images, which can be scripted like preloadCR.sh
  5. Potentially reducing time of a Disaster Recovery rebuild of the platform.
  6. You have HA/DR for your registry. Here is are the docs on IBM Cloud Container Registry HA/DR
  7. You can manage images for most recent or periodic purge.
  8. Vulnerability scans can be done via the service.
- Cons:
  1. If you delete your existing API KEY to the registry, then you need to recreate and link the secrets to the services.
  2. Maintenance if it’s on premises. Backup, vulnerability scans.
  3. Additional operational expense. If on Cloud, you can have OpEx cost which may or may not exceed disk cost from Internal Registry option.
    Note Please double check Service plans egress or pull costs during a load.
  4. Additional steps to set up which are described below.

Creating an External Container Registry on IBM Cloud

Most likely you will have the lite plan which will work fine with an internal registry. All the images are pulled from the IBM Container registry for Cloud Pak for Data and pushed to the internal registry. To use the External Registry option, you will need to upgrade your plan from lite to standard. Standard allows you to pay for storage of containers and not limited to 512MB. It also provides unlimited pull.
Note Please double check Service plans egress or pull costs during a load.

Change plans from Lite to Standard
Adjust the Quotas (if you want)
Select your region. The default is Dallas or us-south
Create a namespace. I used datapartner.
- Using command line ibmcloud cr namespace-add datapartner
Adjust namespace image retention
- Options are all or latest
Create an API Key and remember to download the apikey.json file, so you have the actual key. Once you close it, the key string is unviewable. You will need to create a new one.

Load the External Container Registry on IBM Cloud

You will need to have the following before loading the registry.
- Your IBM Cloud APIKey from the last section
- The URL for your registry
- Namespace for your registry
- Cloud Pak for Data CLI (cpd-cli) installed and its home location.
- A list of assemblies you wish to load.
  Note: If doing this for a paying customer, make sure the list of assemblies is a list that they are entitled for as there is no error handling if there is an authorization error from the container.
Download and edit the preloadCR.sh script.
Note: All the assemblies I included in this script took about 2 hours to load. Think about the time saves as these would be downloaded pushes for each cluster.
You will use the original repo.yaml file when loading the new external repo. Patches are not preloaded.

Using the preloaded Registry

Create a new repo.yaml file. It should not be the same as the default. I’ll call mine repo.icr.yaml
NOTE The IAM_KEY is the APIKEY you created earlier and not the entitlement registry apikey that you used in the preloadCR.sh script or use to use.
```
 registry:
  - url: us.icr.io/<NAMESPACE CREATED ABOVE>
  username: iamapikey
  apikey: <IAM_KEY>
  namespace: ""
  name: base-registry
 fileservers:
 - url: https://raw.github.com/IBM/cloud-pak/master/repo/cpd/3.5
```
You have accepted all the licenses during the preload.
To install run the following commands:

Run env to verify that the following variables are exported

OpenShift 4.x

  export NAMESPACE=zen
  export STORAGE_CLASS=ibmc-file-custom-gold-gid

Set the security aspects for lite to install properly

./cpd-cli adm --repo ./repo.icr.yaml  --namespace ${NAMESPACE} --apply --accept-all-licenses --assembly lite

Deploy lite by running the following: Note: This command also creates secrets to the external registry if they don’t exist. This is helpful to know, if you have your APIKey compromised and need to delete it and create a new one.
```
./cpd-cli install --repo ./repo.icr.yaml --namespace ${NAMESPACE} --storageclass ${STORAGE_CLASS} --insecure-skip-tls-verify  --accept-all-licenses --assembly lite
```
For other services, you will follow the same routine as before described in the table below. The caveat is that you will use the structure of the command just the above command. There is no use of --transfer-image-to=${DOCKER_REGISTRY_PREFIX}/${NAMESPACE} --target-registry-username=ocadmin --target-registry-password=$(oc whoami -t) --cluster-pull-prefix ${LOCAL_REGISTRY}/${NAMESPACE} when using the external registry when installing services.

WORK AROUND: There is an issue with watson-discovery with it’s generated secrets. During the second install, after watson-discovery-operator x86_64 completes, check by running oc get wd wd, you will want to run the patch before the watson-discovery-release x86_64 gets to far. I changed it 3 seconds after it was created. Patch command is oc patch wd wd --type=merge --patch '{"spec": {"shared": {"imagePullSecret": "docker-pull-clonetest-us-icr-io-datapartner-watson-discovery-registry-registry" } } }' Your secret name may vary if you are using different registry. This command oc get secrets | grep 'docker-pull-.*-watson-discovery-registry-registry' | cut -d ' ' -f 1 will help you find the accurate name for the imagePullSecret value.

NOTE: DB2 provision with Oracle compatibility mode, has a UI bug which will make the Console spin and not containers are deployed. Fix available with 3.5.4 GA 3/26/2021

Data Virtualization	Db2 Warehouse Note above	Db2 OLTP Note above
Watson Knowledge Catalog	Watson Studio	Watson Machine Learning
Watson OpenScale Installed, need to test service	Analytics Dashboards	DataStage
Analytics Engine	Cognos Analytics	Watson Discovery WORKAROUND above
EDB PostgreSQL Need to write up	EDB Postres Enterprise Need to write up	Data Management Console Need to write up

Patching the assemblies

Patching is slightly different than installing. You will be transferring the patch to from IBM Registry to the external while patching, so you will use the original repo.yaml file which points off to .

./cpd-cli patch --repo ./repo.yaml --namespace ${NAMESPACE} --transfer-image-to <registry: - url: value from repo.icr.yaml>   --target-registry-username=iamapikey  --target-registry-password=<IAM_KEY>  --insecure-skip-tls-verify --action transfer --assembly lite --patch-name cpd-3.5.2-lite-patch-2

Note before applying cpd-3.5.2-lite-patch-2 read the instructions: It will save you some recovery work.

Upgrading the assemblies

Upgrading is similar to installing, but you don’t need the storageclass parameter. Here is an example of Analytics Dashboard upgrade. Note: You are using the ICR repo file vs the one you use for patching.
```
./cpd-cli upgrade --repo ./repo.icr.yaml --namespace ${NAMESPACE}  --insecure-skip-tls-verify  --accept-all-licenses --assembly cde
```

Applying a new APIKey to the cluster

If you run into the need delete an existing APIKey which you used for the external registry, there are steps to take to recover the cluster. Any new pod pulled from the registry will fail with and ImagePullBackOff

Create the new APIKey and remember to download the apikey.json file, so you have the actual key. Once you close it, the key string is unviewable. You will need to create a new one.
Update the repo.icr.yaml file or whatever you called it.

Delete the existing dockerpull secrets oc get secrets -n ${NAMESPACE}| grep docker-pull | awk '{ print $1 }'

  $ oc get secrets -n ${NAMESPACE} | grep pull | awk '{ print $1 }'
docker-pull-zen-us-icr-io-datapartner-base-registry-registry
  $ oc delete secrets docker-pull-zen-us-icr-io-datapartner-base-registry-registry

Run an install command of a service that is already installed. There is probably a more “OpenShift” way to do this, but too many things to link. This will rebuild the missing secrets and link the services to the secrets. Example:
```
  $ ./cpd-cli install --repo ./repo.icr.yaml --namespace ${NAMESPACE} --storageclass ${STORAGE_CLASS} --insecure-skip-tls-verify  --accept-all-licenses --assembly dmc
```
In another terminal run watch "oc get pods | egrep -v 'Completed|1/1|2/2|3/3|4/4|5/5|6/6|7/7'" and watch OpenShift recover.
If Watson Discovery has not been fixed, you may need to run the workaround.