IBMPartnerDemo

View project on GitHub

Install instructions for Cloud Pak for Data (work in progress)

Knowledge Studio

Industry Accelerators

You may want to start out by looking at the industry accelerators to understand what you will want to install.

Data sets

Here are a few good places to find data sets

The AI Ladder and how Cloud Pak for Data

Clone and Reinstate Clusters

Once you have an environment the way you want it, create a clone of the environment. Save this of to Cloud Object Storage or S3. You can then reinstate this to a cluster with similar attributes. This means if you are using IBM Cloud with Storage Classes of ibmc-file-gold-gid, then you can’t reinstate this to AWS at this moment. There are the instructions to the Clone and Reinstate tool

Migrating from Cloud Pak for Data 3

  1. High level steps:
    • Apply latest patches for all services on cp4d v3.0.1 Note I missed this step and had issues with WML, Db2-OLTP and DV
    • Upgrade OCP v4.3 to V4.5 via v4.4
    • Make sure CPD running fine
    • Upgrade CPD v3.0.1 to v3.5
  2. Set up the Cloud Pak for Data Client
  3. Login to Openshift
  4. Run env to verify that the following variables are exported
    • OpenShift 4.x 
        export NAMESPACE=zen
  export STORAGE_CLASS=ibmc-file-gold-gid
  export DOCKER_REGISTRY_PREFIX=$(oc get routes image-registry -n openshift-image-registry -o template=\{\{.spec.host\}\})
  export LOCAL_REGISTRY=image-registry.openshift-image-registry.svc:5000
  5. Convert the control plane metadata to be compatible with the new version 3.5 model by running the following command: ./cpd-cli operator-upgrade -n ${NAMESPACE}
  6. Update the Control plane or lite assembly. This is a dry run, so you understand what will be updated. This takes about 60 seconds. ./cpd-cli adm --repo ./repo.yaml --namespace ${NAMESPACE} --latest-dependency --accept-all-licenses --assembly lite
  7. Apply the changes. This takes about 90 seconds. ./cpd-cli adm --repo ./repo.yaml --namespace ${NAMESPACE} --latest-dependency --accept-all-licenses --assembly lite --apply
  8. Upgrade the control plane, but start with a dry run first to validate that you have everything in order. ./cpd-cli upgrade --repo ./repo.yaml --namespace ${NAMESPACE} --transfer-image-to=${DOCKER_REGISTRY_PREFIX}/${NAMESPACE} --target-registry-username=ocadmin --target-registry-password=$(oc whoami -t) --cluster-pull-prefix ${LOCAL_REGISTRY}/${NAMESPACE} --insecure-skip-tls-verify --latest-dependency --accept-all-licenses --assembly lite --dry-run
  9. Do the actual upgrade. This took me 26 minutes. ./cpd-cli upgrade --repo ./repo.yaml --namespace ${NAMESPACE} --transfer-image-to=${DOCKER_REGISTRY_PREFIX}/${NAMESPACE} --target-registry-username=ocadmin --target-registry-password=$(oc whoami -t) --cluster-pull-prefix ${LOCAL_REGISTRY}/${NAMESPACE} --insecure-skip-tls-verify --latest-dependency --accept-all-licenses --assembly lite
  10. Check the status of the upgrade. This will give you a list of the other services to update and patch. ./cpd-cli status --repo ./repo.yaml --namespace ${NAMESPACE} --patches --available-updates --assembly lite
  11. Apply security changes for the services that you have installed. I did Data Virtualization. Removing the --apply flag will be a dry run. ./cpd-cli adm --repo ./repo.yaml --namespace ${NAMESPACE} --latest-dependency --accept-all-licenses --assembly dv --apply
  12. Upgrade the services that you have installed. I did Data Virtualization and it took 90 minutes. Note on a few upgrades of services, I have seen the upgrade fail, but when I ran it again it succeeded. Most errors were waiting on an Operator Lock.
    ./cpd-cli upgrade --repo ./repo.yaml --namespace ${NAMESPACE} --transfer-image-to=${DOCKER_REGISTRY_PREFIX}/${NAMESPACE} --target-registry-username=ocadmin --target-registry-password=$(oc whoami -t) --cluster-pull-prefix ${LOCAL_REGISTRY}/${NAMESPACE} --insecure-skip-tls-verify --latest-dependency --accept-all-licenses --assembly service name>
  13. Create a profile to be used when upgrading the services.
  14. Upgrade any running services instances. Each service has its own prerequisites, so please consult the knowledge center.
    List ./cpd-cli service-instance list --profile cpd-admin-profile Upgrade: ./cpd-cli service-instance upgrade --profile cpd-admin-profile --service-type dv --instance-name "DV Instance" --version 1.5.0.0

Provisioning Cloud Pak for Data

Provision the Control plane using IBM Cloud tile

  1. Click Catalog
  2. Click Software
  3. Filter on Analytics or In the search box enter Cloud Pak and press enter. This should list the available Cloud Paks that can be provisioned. In these instructions, you are going to install Cloud Pak for Data.
  4. Click on the Cloud Pak for Data tile.
  5. Click on the Readme tab to review what you can provision using Schematics (Terraform).

Back to Table of Contents

Building the configuration

  1. On the create tab, this is where we will build the configuration of the Cloud Pak for Data clusters
  2. First section describes the bare minimum configuration. For Cloud Pak for Data it is as follows. Each cluster must meet a set of minimum requirements: 3 nodes with 16 cores, 64GB memory, and 25GB disk per node. As I know I want to add Db2 Warehouse and provision a Data Virtualization instance, I will start with 4 node OpenShift cluster. As I add more features, I am needing more capacity when I provision the instance.
  3. Select the cluster that was previously created. Where you created project zen.
  4. Select the project. Click Add Project + and enter something like *zen
  5. Rename the workspace to something you will remember like CPDDemo-HealthCare for a health care demo.
  6. Pick your resource group, if you did not create one, just use Default.
  7. You can tag your environment for easy search. This is not required.
  8. Scroll down and select Run Script if you do not have permissions, you can share a link and have an administrator execute this.
  9. Scroll down and select the services that you desire to add to Cloud Pak for Data. Note: This is a first time set up action only. Currently, Once provisioned, toggling true then false will lead to no further action. You will need to uninstall or install again, which will be shown later. To install all of these, you will need significantly more infrastructure. The Master or controller OCP Nodes are part of a Service Account. The Worker or Compute Nodes are added to your account. VLAN Spanning is needed for this to work. To get a better idea for sizing infrastructure, you can used this Sales Config tool. It may require an IBM approval for access, which you will use your IBM ID.
    • AI (Apache Spark, IBM Streams, RStudio Server, Watson Studio, Watson Machine Learning, Watson OpenScale)
    • Dashboarding (Analytic Dashboards)
    • Data Governance (Watson Knowledge Catalog, Open Source Management)
    • Data Sources (Data Virtualization, Db2 Warehouse).
  10. On the right, you should see your entitlements and a yellow box, which will turn green after you run the script below.
  11. Check the box and to accept the license.
  12. Once install is complete, you may need to check for and apply patches. Hopefully in the future you will always pull the latest from the container registry.

Back to Table of Contents

Review Services installed

  1. From the Schematics workspace, Click Offering dashboard
  2. Depending on your browser, you will need to accept the certificate. General path is Advanced then accept risk. This will bring you to the login page. Bookmark this for future usage.
  3. Login using default credentials as seen in video.
  4. Along the black menu bar at the top the icon to the right on the magnifying glass is Services. Click on this link.
  5. Take a look at the services that are enabled. These have been installed or deployed. Depending on what was selected any of these could be available or enabled.
    • AI (Apache Spark, IBM Streams, RStudio Server, Watson Studio, Watson Machine Learning, Watson OpenScale)
    • Dashboarding (Analytic Dashboards)
    • Data Governance (Watson Knowledge Catalog, Open Source Management)
    • Data Sources (Data Virtualization, Db2 Warehouse).
  6. Some are set up automatically and some you need to provision.

Back to Table of Contents

Provision the Control plane using command line

Setting up the Cloud Pak for Data Client

  1. You have already set up the client environment. If not go have to the first page and execute these step under Installing the client environment You will have logged into the OpenShift cluster, set the project to zen already built the encrypted route to the internal container repository. This is where all of your containers and helm charts will be stored. If zen doesn’t exist, then your probably went with default.
  2. Here is a link to the knowledge center for Cloud Pak for Data installation instructions. The next steps are a sequential version for this environment.
  3. How do I get the installer? Click the CPD Github to download the Enterprise Edition. Named cpd-cli-darwin-EE-3.5.3.tgz. File is about 101MB.
  4. Unarchive the client bits. You should find the following.
    • cpd-cli
    • repo.yaml
    • plugins (directory)
    • LICENSES (directory)
  5. Find the file called repo.yaml This is a simple file, but key. Here is where you will add the apikey to get access to the container registry for the additional Services. There are services where you will need to access different container repository namespaces I will include a repo.product.yaml when I document the install.
  6. In a web browser, Login and access the entitlement registry. Click Get entitlement key on the left side above Library. Either Generate a key or Copy the existing key. This will populate your scratchpad where you can paste it into the repo.yaml file.
  7. Paste the apikey into the Yaml file to the right of the apikey: After you paste, it should something like this only longer. Note Leave namespace as is as I have not yet found a use for it. 
      ---
  fileservers:
 -
   url: "https://raw.github.com/IBM/cloud-pak/master/repo/cpd/3.5"
  registry:
 -
   apikey: eyJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJJQk0gTWFya2V0cGxhY2UiLCJpYXQiOjE1NzYwMjA4NTAsImp0aSI6ImViMzliZTk4MDJjNzRjYjM4M2Q3YjYxYTYzNmUwY2VkIn0.5Czky1_pLPLIEzScJzQ_qVNxo3v3EXyVGK7a7Dafcrg
   name: base-registry
   namespace: ""
   url: cp.icr.io/cp/cpd
   username: cp
  8. Run the client command to verify that it works. On Mac you will need to verify the command by launching it from finder. 
     ./cpd-cli

    NOTE: cpd-cli will run different operators to execute commands. These operators or executors are under lib/os name These are config, cpd-base, cpdbr, cpdtool, and service-instance. On Mac, you may see a sercurity pop-up running something like ./cpd-cli status --repo ./repo.yaml --namespace zen --assembly lite --patches --available-updates You need to tell Mac that you approve to run these. The simplest way is to run xattr -c cpd-cli against the executable files above. This will clear all the attributes from the file. Another way is to open Finder and move to the executors and open in Terminal and accept Open. This tell the system to allow you to run these. There may be something similar on Windows, but I have not tried this.

Back to Table of Contents

Preparing the cluster

Prepare the nodes and storage

Running Scripts to prepare file sizes and storage classes. Coming soon

Back to Table of Contents

Choosing Internal vs External Image Registry

Choices about and setting up an External Container Image Registry. By default, you will use the internal registry. This is not considered an industry practice in Red Hat circles. They suggest Internal is mearly a cache and you should set up a more robust External Registry.

Back to Table of Contents

Installing the Cloud Pak for Data Control Plane by command line

Note: This is not a recommended option on IBM Cloud. Since possible, I am documenting.

  1. Follow the instructions in Setting up the Cloud Pak for Data Client
  2. Follow the instructions in Retrieving the token to log into OpenShift on IBM Cloud
  3. Create your OpenShift namespace or project that you will install Cloud Pak for Data.
    • Example: oc project zen
  4. Run env to verify that the following variables are exported
    • OpenShift 4.x 
        export NAMESPACE=zen
  export STORAGE_CLASS=ibmc-file-gold-gid
  export DOCKER_REGISTRY_PREFIX=$(oc get routes image-registry -n openshift-image-registry -o template=\{\{.spec.host\}\})
  export LOCAL_REGISTRY=image-registry.openshift-image-registry.svc:5000
  5. Make sure all the node settings are set on the worker nodes. All this is done for you Provision the Control plane using IBM Cloud tile or run 
    ./cpd-cli adm --repo ./repo.yaml  --namespace ${NAMESPACE} --apply --accept-all-licenses --assembly lite
  6. Install the control plane or the lite assembly 
      ./cpd-cli install --repo ./repo.yaml --namespace ${NAMESPACE} --storageclass ${STORAGE_CLASS} --transfer-image-to=${DOCKER_REGISTRY_PREFIX}/${NAMESPACE} --target-registry-username=ocadmin  --target-registry-password=$(oc whoami -t) --cluster-pull-prefix ${LOCAL_REGISTRY}/${NAMESPACE} --insecure-skip-tls-verify --latest-dependency --accept-all-licenses --assembly lite
  7. Verify the installation 
      ./cpd-cli status --namespace ${NAMESPACE} --assembly lite
  8. Check for patches 
      ./cpd-cli status  --repo ./repo.yaml --namespace ${NAMESPACE} --patches --available-updates --assembly lite
  9. If there are patches apply the highest number as it will be cumulative. Some patches have prerequisite patches because they have dependencies on another service or on a set of shared, common services. If the patch details list one or more prerequisite patches, you must install the prerequisite patches before you install the service patch. You can run the following command to determine whether any of the prerequisite patches are already installed on the cluster:

Back to Table of Contents

How can I patch a service or control plane

From time to time any software needs a patch for security reasons, new feature or a bug fix. How do you know that there is a patch for a specifica service, common services or the control plane. Take a look here for v3.0.x. This document has a link to each of the service patches and any extra work that might be needed. Most patches need a prerequisite patch for the common services.

  1. Run env to verify that the following variables are exported
    • OpenShift 4.x 
      export NAMESPACE=zen
export STORAGE_CLASS=ibmc-file-gold-gid
export DOCKER_REGISTRY_PREFIX=$(oc get routes image-registry -n openshift-image-registry -o template=\{\{.spec.host\}\})
export LOCAL_REGISTRY=image-registry.openshift-image-registry.svc:5000
  2. Check for patches 
      ./cpd-cli status  --repo ./repo.yaml --namespace ${NAMESPACE} --patches --available-updates --assembly lite
  3. Run the command to patch the common services. Notice that the command is patch, patch-name is cpd-3.5.2-lite-patch-1. This name will change after this writing. Note the assembly name can be lite, wkc or wsl. 
     ./cpd-cli patch --repo ./repo.yaml  --namespace ${NAMESPACE}  --transfer-image-to=${DOCKER_REGISTRY_PREFIX}/${NAMESPACE} --target-registry-username=ocadmin  --target-registry-password=$(oc whoami -t) --cluster-pull-prefix ${LOCAL_REGISTRY}/${NAMESPACE}  --insecure-skip-tls-verify --action transfer --assembly lite --patch-name cpd-3.5.2-lite-patch-2
  4. Verify that the patch has been applied. 
     Toms-MBP:~ tjm$ oc project zen
 Already on project "zen" on server "https://c106-e.us-south.containers.cloud.ibm.com:31432".
 Toms-MBP:~ tjm$ oc describe cpdinstall cr-cpdinstall | grep "Patch Name:" | sort | uniq | cut -d: -f2
    cpd-3.5.2-lite-patch-2
  5. you can repeat this pattern, replacing the values to the right of assembly and patch-name
  6. If something goes wrong and you need to roll back a patched assembly, this is the command: 
     ./cpd-cli patch rollback  --namespace ${NAMESPACE}   --cluster-pull-username=ocadmin  --cluster-pull-password=$(oc whoami -t) --cluster-pull-prefix ${LOCAL_REGISTRY}/${NAMESPACE}  --assembly lite --patch-name cpd-3.5.2-lite-patch-4

Back to Table of Contents

To enable or disable the default admin users

  1. It is suggested to connect your user repository to an LDAP system.
  2. Once connected to LDAP, you should disable the default admin user.
  3. To do this you will run a script in the user management pod. 
      export NAMESPACE=zen
  oc exec -it -n $NAMESPACE  $(oc get pod -n $NAMESPACE  -l component=usermgmt | tail -1 | cut -f1 -d\ ) -- bash -c "/usr/src/server-src/scripts/manage-user.sh --disable-user admin"
  4. To reenable admin user run the following, which will prompt you for a new password: 
      export NAMESPACE=zen
  oc exec -it -n $NAMESPACE  $(oc get pod -n $NAMESPACE  -l component=usermgmt | tail -1 | cut -f1 -d\ ) -- bash -c "/usr/src/server-src/scripts/manage-user.sh --enable-user admin"

    Note: If you have lost or forgotten your CPD Admin password, you can log into OpenShift then execute the enable command which prompts you for a new password.

Back to Table of Contents

Troubleshooting and managing Cloud Pak for Data through the console

Manage Deployments

coming soon
Back to Table of Contents

Manage Users

  1. From the Navigation Menu, scroll down to Administration then User Management
  2. Click New User +
  3. Complete the form and select the roles you wish the user to have. This is a good opportunity to create different roles so that you can get a good idea of what each role has what capability.
  4. Click Create
  5. If you need to add the user to certain instances that you desire them to have access. These services such as Data Virtualization, Cognos, Planning Analytics, Db2, DB2 Warehouse, Spark ,etc.
    • From the Navigation Menu, scroll down to Services then Instances
    • On the right, there are 3 vertical dots. Click this and select Manage Access
    • Each service has similar look and feel, but different roles.

Back to Table of Contents

Viewing logs

coming soon
Back to Table of Contents

## Data Virtualization ### Install Data Virtualization

  1. Run env to verify that the following variables are exported - OpenShift 4.x 
    export NAMESPACE=zen
export STORAGE_CLASS=ibmc-file-gold-gid
export DOCKER_REGISTRY_PREFIX=$(oc get routes image-registry -n openshift-image-registry -o template=\{\{.spec.host\}\})
export LOCAL_REGISTRY=image-registry.openshift-image-registry.svc:5000
  2. Set the security aspects for Data Virtualization to install properly ~~~ ./cpd-cli adm –repo ./repo.yaml –namespace ${NAMESPACE} –apply –accept-all-licenses –assembly dv ~~~
  3. Deploy Data Virtualization by running the following: ~~~ ./cpd-cli install –repo ./repo.yaml –namespace ${NAMESPACE} –storageclass ${STORAGE_CLASS} –transfer-image-to=${DOCKER_REGISTRY_PREFIX}/${NAMESPACE} –target-registry-username=ocadmin –target-registry-password=$(oc whoami -t) –cluster-pull-prefix ${LOCAL_REGISTRY}/${NAMESPACE} –insecure-skip-tls-verify –latest-dependency –accept-all-licenses –assembly dv ~~~
  4. Verify the installation 
    ./cpd-cli status --namespace ${NAMESPACE} --assembly dv
    1. Check for patches 
      ./cpd-cli status  --repo ./repo.yaml --namespace ${NAMESPACE} --patches --available-updates --assembly dv
    2. If there are patches apply the highest number as it will be cumulative. Some patches have prerequisite patches because they have dependencies on another service or on a set of shared, common services. If the patch details list one or more prerequisite patches, you must install the prerequisite patches before you install the service patch. You can run the following command to determine whether any of the prerequisite patches are already installed on the cluster:

Back to Table of Contents

Data Management Console

Install Data Management Console

  1. Run env to verify that the following variables are exported
    • OpenShift 4.x 
      export NAMESPACE=zen
export STORAGE_CLASS=ibmc-file-gold-gid
export DOCKER_REGISTRY_PREFIX=$(oc get routes image-registry -n openshift-image-registry -o template=\{\{.spec.host\}\})
export LOCAL_REGISTRY=image-registry.openshift-image-registry.svc:5000
  2. Set the security aspects for Data Management Console to install properly 
       ./cpd-cli adm --repo ./repo.yaml  --namespace ${NAMESPACE} --apply --accept-all-licenses --assembly dmc
  3. Deploy Data Virtualization by running the following: 
       ./cpd-cli install --repo ./repo.yaml --namespace ${NAMESPACE} --storageclass ${STORAGE_CLASS} --transfer-image-to=${DOCKER_REGISTRY_PREFIX}/${NAMESPACE} --target-registry-username=ocadmin  --target-registry-password=$(oc whoami -t) --cluster-pull-prefix ${LOCAL_REGISTRY}/${NAMESPACE} --insecure-skip-tls-verify --latest-dependency --accept-all-licenses --assembly dmc
  4. Verify the installation 
    ./cpd-cli status --namespace ${NAMESPACE} --assembly dmc
    1. Check for patches 
      ./cpd-cli status  --repo ./repo.yaml --namespace ${NAMESPACE} --patches --available-updates --assembly dmc
    2. If there are patches apply the highest number as it will be cumulative. Some patches have prerequisite patches because they have dependencies on another service or on a set of shared, common services. If the patch details list one or more prerequisite patches, you must install the prerequisite patches before you install the service patch. You can run the following command to determine whether any of the prerequisite patches are already installed on the cluster:

Back to Table of Contents

Set up Data Virtualization

  1. Click on the Services icon to get to the services catalog.
  2. Go to the Data Virtualization tile, you see to “Provision Instance”.
  3. Click on Provision Instance.
  4. Then Click Configure. Check the box to the left of Set up semaphore parameters for Data Virtualization automatically.
  5. Click Configure.
  6. Select the number of nodes and cores to allocate. For a demo system defaults are enough. Note: Depending on how many node you have in your worker pool, this could fail provisioning with a not enough resource message. If you configure a default 3 16/64 workers and install all of the services, this message can appear. Resizing your worker pool to 4 will get you around this.
  7. Make sure your change the storage classes from default to ibmc-file-gold-gid. Choosing default could cause a failure.
  8. Once provisioned, set some users to have Data Engineer role. - Go to the Navigator on the left and scroll down to Administer Click on Manage Users. - On the right of the list of users, you will see a pencil, which indicates edit. Click the pencil for the user you desire to edit. If you pick admin, you will see this is a new role added based on the Data Virtualization service.
  9. Go to My Instance
  10. Once complete, you will see Data Virtualization under the Collect area on the left table of contents. From here you can add data sources or pointers to file folders. Note: I have not yet tested this with Remote Data Connections aka local file folders.

Note: if you delete the instance, you might get a PVC in terminating state. Execute the following to clean this up. kubectl patch pvc dv-pvc -p '{"metadata":{"finalizers":null}}'

What can be done with Data Virtualization and Watson Knowledge Catalog

Back to Table of Contents

Quick Tutorial on Data Virtualization

Here is a quick 35 minute tutorial on using Data Virtualization on Cloud Pak for Data.

Back to Table of Contents

Watson Studio

Install Watson Studio

Coming Soon Need to add override file

  1. Run env to verify that the following variables are exported
    • OpenShift 4.x 
      export NAMESPACE=zen
export STORAGE_CLASS=ibmc-file-gold-gid
export DOCKER_REGISTRY_PREFIX=$(oc get routes image-registry -n openshift-image-registry -o template=\{\{.spec.host\}\})
export LOCAL_REGISTRY=image-registry.openshift-image-registry.svc:5000
  2. Set the security aspects for Watson Studio to install properly 
       ./cpd-cli adm --repo ./repo.yaml  --namespace ${NAMESPACE} --apply --accept-all-licenses --assembly wsl
  3. Deploy Watson Studio by running the following: 
       ./cpd-cli install --repo ./repo.yaml --namespace ${NAMESPACE} --storageclass ${STORAGE_CLASS} --transfer-image-to=${DOCKER_REGISTRY_PREFIX}/${NAMESPACE} --target-registry-username=ocadmin  --target-registry-password=$(oc whoami -t) --cluster-pull-prefix ${LOCAL_REGISTRY}/${NAMESPACE} --insecure-skip-tls-verify --latest-dependency --accept-all-licenses --assembly wsl
  4. Verify the installation 
    ./cpd-cli status --namespace ${NAMESPACE} --assembly wsl
    1. Check for patches 
      ./cpd-cli status  --repo ./repo.yaml --namespace ${NAMESPACE} --patches --available-updates --assembly wsl
    2. If there are patches apply the highest number as it will be cumulative. Some patches have prerequisite patches because they have dependencies on another service or on a set of shared, common services. If the patch details list one or more prerequisite patches, you must install the prerequisite patches before you install the service patch. You can run the following command to determine whether any of the prerequisite patches are already installed on the cluster:

Back to Table of Contents

Set up Watson Studio

  1. If you pick Watson Studio or Watson Machine Learning tiles, there will be no actions to take. You can proceed and create a project.

Back to Table of Contents

Increase the capacity or scale up

  1. To scale services, you will need to have the command line installed. (Instruction under the Adding additional services section)
  2. You can scale up the services by executing these commands for either wsl. Note: There is currently no scale down function.
  3. Run env to verify that the following variables are exported
    • OpenShift 4.x 
      export NAMESPACE=zen
export STORAGE_CLASS=ibmc-file-gold-gid
export DOCKER_REGISTRY_PREFIX=$(oc get routes image-registry -n openshift-image-registry -o template=\{\{.spec.host\}\})
export LOCAL_REGISTRY=image-registry.openshift-image-registry.svc:5000
  4. Scale up Watson Studio by running the following (config sizes are small, medium and large): 
    ./cpd-cli scale -n $NAMESPACE --config medium  --load-from ./cpd-cli-workspace  -a wsl
  5. Scale down Watson Studio by running the following (config sizes are small, medium and large): 
    ./cpd-cli scale -n zen --config small  --load-from ./cpd-cli-workspace -a wsl

Back to Table of Contents

Watson Machine Learning

Install Watson Machine Learning

Coming Soon Need to add override file

  1. Run env to verify that the following variables are exported
    • OpenShift 4.x 
         export NAMESPACE=zen
   export STORAGE_CLASS=ibmc-file-gold-gid
   export DOCKER_REGISTRY_PREFIX=$(oc get routes image-registry -n openshift-image-registry -o template=\{\{.spec.host\}\})
   export LOCAL_REGISTRY=image-registry.openshift-image-registry.svc:5000
  2. Set the security aspects for Watson Machine Learning to install properly 
       ./cpd-cli adm --repo ./repo.yaml  --namespace ${NAMESPACE} --apply --accept-all-licenses --assembly wml
  3. Deploy Watson Machine Learning by running the following: 
       ./cpd-cli install --repo ./repo.yaml --namespace ${NAMESPACE} --storageclass ${STORAGE_CLASS} --transfer-image-to=${DOCKER_REGISTRY_PREFIX}/${NAMESPACE} --target-registry-username=ocadmin  --target-registry-password=$(oc whoami -t) --cluster-pull-prefix ${LOCAL_REGISTRY}/${NAMESPACE} --insecure-skip-tls-verify --latest-dependency --accept-all-licenses --assembly wml
  4. Verify the installation 
       ./cpd-cli status --namespace ${NAMESPACE} --assembly wml
  5. Check for patches 
       ./cpd-cli status  --repo ./repo.yaml --namespace ${NAMESPACE} --patches --available-updates --assembly wml
  6. If there are patches apply the highest number as it will be cumulative. Some patches have prerequisite patches because they have dependencies on another service or on a set of shared, common services. If the patch details list one or more prerequisite patches, you must install the prerequisite patches before you install the service patch. You can run the following command to determine whether any of the prerequisite patches are already installed on the cluster:

Back to Table of Contents

Increase the capacity or scale up your Watson Machine Learning service

  1. To scale services, you will need to have the command line installed. (Instruction under the Adding additional services section)
  2. You can scale up the services by executing these commands for either wml. Note: There is currently no scale down function.
  3. Run env to verify that the following variables are exported .
  • OpenShift 4.x 
     export NAMESPACE=zen
 export STORAGE_CLASS=ibmc-file-gold-gid
 export DOCKER_REGISTRY_PREFIX=$(oc get routes image-registry -n openshift-image-registry -o template=\{\{.spec.host\}\})
 export LOCAL_REGISTRY=image-registry.openshift-image-registry.svc:5000
    1. Scale up Watson Machine Learning by running the following (config sizes are small, medium and large): 
       ./cpd-cli scale -n $NAMESPACE --config medium  --load-from ./cpd-cli-workspace  -a wml
    2. Scale down Watson Machine Learning by running the following (config sizes are small, medium and large): 
       ./cpd-cli scale -n zen --config small  --load-from ./cpd-cli-workspace -a wml

    Back to Table of Contents

Watson OpenScale

Install Watson OpenScale

  1. When setting up OpenScale, there are 2 pre-prerequisites.
    • Watson Machine Learning (Local or remote)
    • Local then install
    • Remote then document the following parameters:
    • Db2 Database (Db2 Advanced or DB2 Warehouse Local or remote)
    • If no remote Db2 instance available, then install either Db2 Advanced or Db2 Warehouse
  2. Run env to verify that the following variables are exported
    • OpenShift 4.x 
        export NAMESPACE=zen
  export STORAGE_CLASS=ibmc-file-gold-gid
  export DOCKER_REGISTRY_PREFIX=$(oc get routes image-registry -n openshift-image-registry -o template=\{\{.spec.host\}\})
  export LOCAL_REGISTRY=image-registry.openshift-image-registry.svc:5000
  3. Set the security aspects for Watson Machine Learning to install properly 
      ./cpd-cli adm --repo ./repo.yaml  --namespace ${NAMESPACE} --apply --accept-all-licenses --assembly aiopenscale
  4. Deploy Watson OpenScale by running the following: 
      ./cpd-cli install --repo ./repo.yaml --namespace ${NAMESPACE} --storageclass ${STORAGE_CLASS} --transfer-image-to=${DOCKER_REGISTRY_PREFIX}/${NAMESPACE} --target-registry-username=ocadmin  --target-registry-password=$(oc whoami -t) --cluster-pull-prefix ${LOCAL_REGISTRY}/${NAMESPACE} --insecure-skip-tls-verify --latest-dependency --accept-all-licenses --assembly aiopenscale
  5. Verify the installation 
      ./cpd-cli status --namespace ${NAMESPACE} --assembly aiopenscale
  6. Check for patches 
     ./cpd-cli status  --repo ./repo.yaml --namespace ${NAMESPACE} --patches --available-updates --assembly aiopenscale
  7. If there are patches apply the highest number as it will be cumulative. Some patches have prerequisite patches because they have dependencies on another service or on a set of shared, common services. If the patch details list one or more prerequisite patches, you must install the prerequisite patches before you install the service patch. You can run the following command to determine whether any of the prerequisite patches are already installed on the cluster:

    Back to Table of Contents

Set up Watson OpenScale

  1. For Watson OpenScale Service, there is an Open button which launch a UI to help configure the initial information.
  2. Let’s provision OpenScale.
  3. You will be met with an auto setup to configure.
  4. First step is accept the local WML instance.
  5. Enter connection details for a Db2 Database. I am using a DB2 Warehouse that I installed later on. You need to have Db2/Db2 Warehouse running somewhere to get OpenScale to work. If you have one handy, you can use this otherwise we will configure it in the next section. Coming soon

NOTE: If you happen to change the admin password, then you need to update a secret

Back to Table of Contents

Db2 Warehouse

Install Db2 Warehouse (SMP)

  1. The first thing you will want to do is to pick one node that will house Db2 Warehouse and add a label.
  2. Run: oc get nodes This will produce a list of nodes.
  3. Run oc describe node | grep -eHostname: -e 'cpu ' -e 'memory ' Review the output and find the node with the least amount of resources allocated You will be setting a label to create Node Affinity. Db2 Warehouse pods will be installed to this particular node. Database will also be provisioned to this node. Sometimes you may need to resize your OpenShift worker pool size to increase capacity. Base on review I will pick node 10.95.7.49 as it has the least amount of resources allocated right now. 
    Toms-MBP:bin tjm$ oc describe node |  grep -eHostname: -e 'cpu ' -e 'memory '
 MemoryPressure   False   Wed, 10 Jun 2020 16:21:28 -0400   Mon, 08 Jun 2020 10:19:26 -0400   KubeletHasSufficientMemory   kubelet has sufficient memory available
 Hostname:    10.95.7.21
 cpu       14941m (94%)      63645m (400%)
 memory    44715538Ki (74%)  150095560704 (244%)
 MemoryPressure   False   Wed, 10 Jun 2020 16:21:29 -0400   Mon, 08 Jun 2020 10:19:13 -0400   KubeletHasSufficientMemory   kubelet has sufficient memory available
 Hostname:    10.95.7.23
 cpu       13960m (87%)      39825m (250%)
 memory    42889746Ki (71%)  106114088960 (172%)
 MemoryPressure   False   Wed, 10 Jun 2020 16:21:32 -0400   Mon, 08 Jun 2020 10:16:15 -0400   KubeletHasSufficientMemory   kubelet has sufficient memory available
 Hostname:    10.95.7.49
 cpu       10718m (67%)      33863m (213%)
 memory    24435218Ki (40%)  85963040Ki (143%)
 MemoryPressure   False   Wed, 10 Jun 2020 16:21:35 -0400   Mon, 08 Jun 2020 10:21:40 -0400   KubeletHasSufficientMemory   kubelet has sufficient memory available
 Hostname:    10.95.7.6
 cpu       11970m (75%)      49570m (312%)
 memory    30833170Ki (51%)  87313121280 (142%)
  4. Bind Db2 Warehouse and provisioned instances to a specific node, by adding a label of icp4data=database-db2wh to a node. Select this node according to known resources available. I picked node 10.95.7.49 from the last command. 
    oc label node <node name or IP Address> icp4data=database-db2wh

    Note: If you resize your OpenShift cluster’s worker pool to a lower number of nodes, it is possible that the node with the label for Db2 Warehouse may be deleted. This would render any database instances unusable until you label another node and restart the db2wh pods, so they start on the same node. Do not try to label 2 nodes as you will get an anti-affinity error in the db2u-0 and unified-console pods, They will stay in a state of pending.

  5. Run env to verify that the following variables are exported
    • OpenShift 4.x 
       export NAMESPACE=zen
 export STORAGE_CLASS=ibmc-file-gold-gid
 export DOCKER_REGISTRY_PREFIX=$(oc get routes image-registry -n openshift-image-registry -o template=\{\{.spec.host\}\})
 export LOCAL_REGISTRY=image-registry.openshift-image-registry.svc:5000
  6. Set the security aspects for Db2 Warehouse to install properly 
      ./cpd-cli adm --repo ./repo.yaml  --namespace ${NAMESPACE} --apply --accept-all-licenses --assembly db2wh
  7. Deploy Db2 Warehouse by running the following: 
      ./cpd-cli install --repo ./repo.yaml --namespace ${NAMESPACE} --storageclass ${STORAGE_CLASS} --transfer-image-to=${DOCKER_REGISTRY_PREFIX}/${NAMESPACE} --target-registry-username=ocadmin  --target-registry-password=$(oc whoami -t) --cluster-pull-prefix ${LOCAL_REGISTRY}/${NAMESPACE} --insecure-skip-tls-verify --latest-dependency --accept-all-licenses --assembly db2wh
  8. This will take some time to download, push to the registry, request new storage from IBM Cloud and provision the services and pods.
  9. Verify the installation 
    ./cpd-cli status --namespace ${NAMESPACE} --assembly db2wh
  10. Check for patches 
    ./cpd-cli status  --repo ./repo.yaml --namespace ${NAMESPACE} --patches --available-updates --assembly db2wh
  11. If there are patches apply the highest number as it will be cumulative. Some patches have prerequisite patches because they have dependencies on another service or on a set of shared, common services. If the patch details list one or more prerequisite patches, you must install the prerequisite patches before you install the service patch. You can run the following command to determine whether any of the prerequisite patches are already installed on the cluster:

Note: Actual time taken is 16 minutes

Back to Table of Contents

Provision a Database instance

  1. Once installed and all pods are up, you can go to the service catalog page with the square with petals icon in upper right.
  2. On the services page, Click the left side filter to go to Datasources to get to Db2 Warehouse tile.
  3. Click the 3 vertical dots on upper left of the tile or Click through the tile then Click Provision Instance.
  4. On the Configure page keep defaults or adjust if you know you need more. Click Next.
  5. On the Storage page, Select Create new storage; Change Storage Class to ibmc-file-gold-gid ; Adjust the size to reflect the amount needed. Default is 100GB.
  6. Click Next
  7. Review the settings. Here you can change the Display name to something more memorable. Click Create.
  8. Your instance is being created and will be accessible from the Services > Db2 Warehouse tile. Also accessed from the Left menu My Instance then Click Provisioned instances
  9. From Provision Instances on the left you will see 3 horizontal dots. From this view, you can watch the steps of the provision, just incase it fails based on insufficient resources. Note: You can see a red triangle that states Failed. This is temporary and is most likely the Cloud provision service waiting on a storage volume to come online and available to mount as a persistent volume to map the persistent volume claim.
  10. Click this and see the options.
    • Open will open the DB2 Warehouse instance for use.
    • View details will provide you the details including user ; password ; jdbc url
    • Manage Access let you add user ids and assign them Admin or User roles.
    • Delete This will delete this particular instance. Note Total time took about 12 minutes

Back to Table of Contents

Create a Table

  1. When you Open the interface, you will run as the user id that provisioned it, for example user999 which is admin, so any tables that are created, by admin, via SQL editor by default will go under Schema user999.
    • You can use the UI and select and create tables under certain schemas.

Back to Table of Contents

Configure db2audit

Here are the steps to configure DB2 Audit in Db2 Warehouse or DB2 OLTP

Back to Table of Contents

Uninstalling DB2 Warehouse.

  1. First you will want to release any storage and delete instances to make sure storage is released. Use Delete to do this.
  2. From the command line:
  3. Set namespace. My namespace is zen your may be different like default
  4. Run env to verify that the following variables are exported
    • OpenShift 4.x 
        export NAMESPACE=zen
  5. Do a dry run uninstall to check what will be taken off. 
      ./cpd-cli uninstall --namespace ${NAMESPACE} --assembly db2wh --uninstall-dry-run
  6. Run the uninstall 
      ./cpd-cli uninstall --namespace ${NAMESPACE}  --assembly db2wh
  7. Go to the Services catalog and verify that Db2 Warehouse is no longer enabled.

Back to Table of Contents

Db2 Advanced

Install Db2 Advanced

  1. The first thing you will want to do is to pick one node that will house Db2 Advanced and add a label.
  2. Run: oc get nodes This will produce a list of nodes.
  3. Run oc describe node | grep -eHostname: -e 'cpu ' -e 'memory ' Review the output and find the node with the least amount of resources allocated You will be setting a label to create Node Affinity. Db2 OLTP pods will be installed to this particular node. Database will also be provisioned to this node. Sometimes you may need to resize your OpenShift worker pool size to increase capacity. Base on review I will pick node 10.95.7.49 as it has the least amount of resources allocated right now. 
     Toms-MBP:bin tjm$ oc describe node |  grep -eHostname: -e 'cpu ' -e 'memory '
  MemoryPressure   False   Wed, 10 Jun 2020 16:21:28 -0400   Mon, 08 Jun 2020 10:19:26 -0400   KubeletHasSufficientMemory   kubelet has sufficient memory available
  Hostname:    10.95.7.21
  cpu       14941m (94%)      63645m (400%)
  memory    44715538Ki (74%)  150095560704 (244%)
  MemoryPressure   False   Wed, 10 Jun 2020 16:21:29 -0400   Mon, 08 Jun 2020 10:19:13 -0400   KubeletHasSufficientMemory   kubelet has sufficient memory available
  Hostname:    10.95.7.23
  cpu       13960m (87%)      39825m (250%)
  memory    42889746Ki (71%)  106114088960 (172%)
  MemoryPressure   False   Wed, 10 Jun 2020 16:21:32 -0400   Mon, 08 Jun 2020 10:16:15 -0400   KubeletHasSufficientMemory   kubelet has sufficient memory available
  Hostname:    10.95.7.49
  cpu       10718m (67%)      33863m (213%)
  memory    24435218Ki (40%)  85963040Ki (143%)
  MemoryPressure   False   Wed, 10 Jun 2020 16:21:35 -0400   Mon, 08 Jun 2020 10:21:40 -0400   KubeletHasSufficientMemory   kubelet has sufficient memory available
  Hostname:    10.95.7.6
  cpu       11970m (75%)      49570m (312%)
  memory    30833170Ki (51%)  87313121280 (142%)
  4. Bind Db2 Advanced and provisioned instances to a specific node, by adding a label of icp4data=database-db2-oltp to a node. Select this node according to known resources available. I picked node 10.95.7.49 from the last command. 
     oc label node <node name or IP Address> icp4data=database-db2oltp
  5. You can dedicate one or more worker nodes to your Db2® database service. Do this before installing the service.
  6. Run env to verify that the following variables are exported
    • OpenShift 4.x 
        export NAMESPACE=zen
  export STORAGE_CLASS=ibmc-file-gold-gid
  export DOCKER_REGISTRY_PREFIX=$(oc get routes image-registry -n openshift-image-registry -o template=\{\{.spec.host\}\})
  export LOCAL_REGISTRY=image-registry.openshift-image-registry.svc:5000
  7. Set the security aspects for Db2 Advanced to install properly 
    ./cpd-cli adm --repo ./repo.yaml  --namespace ${NAMESPACE} --apply --accept-all-licenses --assembly db2oltp
  8. Deploy Db2 Advanced by running the following: 
    ./cpd-cli install --repo ./repo.yaml --namespace ${NAMESPACE} --storageclass ${STORAGE_CLASS} --transfer-image-to=${DOCKER_REGISTRY_PREFIX}/${NAMESPACE} --target-registry-username=ocadmin  --target-registry-password=$(oc whoami -t) --cluster-pull-prefix ${LOCAL_REGISTRY}/${NAMESPACE} --insecure-skip-tls-verify --latest-dependency --accept-all-licenses --assembly db2oltp
  9. This will take some time to download, push to the registry, request new storage from IBM Cloud and provision the services and pods.
  10. The deployed service will look like this from oc get pods. The completed pod is a load job which can be deleted. 
    db2oltp-catalog-11530-6c488d895f-jddzn   1/1       Running     0          5m11s
db2oltp-catalog-11530-uploadjob-hcsqr    0/1       Completed   0          5m11s
  11. Verify the installation 
    ./cpd-cli status --namespace ${NAMESPACE} --assembly db2oltp
  12. Check for patches 
    ./cpd-cli status  --repo ./repo.yaml --namespace ${NAMESPACE} --patches --available-updates --assembly db2oltp
  13. If there are patches apply the highest number as it will be cumulative. Some patches have prerequisite patches because they have dependencies on another service or on a set of shared, common services. If the patch details list one or more prerequisite patches, you must install the prerequisite patches before you install the service patch. You can run the following command to determine whether any of the prerequisite patches are already installed on the cluster:

    Back to Table of Contents

Provision an OLTP Database instance

  1. Once installed and the db2oltp-catalog-11530 pod is running or the service tile is marked as available , you can go to the service catalog page with the square with petals icon in upper right.
  2. On the services page, Click the left side filter to go to Datasources to get to Db2 Advanced tile.
  3. Click the 3 vertical dots on upper left of the tile or Click through the tile then Click Provision Instance.
  4. On the Configure page keep defaults or adjust if you know you need more. Click Next. NOTE Make sure the Value for node label matches what was set above.
  5. On the System Storage page, Change Storage Class to ibmc-file-gold-gid ; Adjust the size to reflect the amount needed. Default is 100GB.
  6. Click Next
  7. On the User Storage page, Change Storage Class to ibmc-file-gold-gid ; Adjust the size to reflect the amount needed. Default is 100GB.
  8. Click Next
  9. On the Backup Storage page, Change Storage Class to ibmc-file-gold-gid ; Adjust the size to reflect the amount needed. Default is 100GB.
  10. Click Next
  11. Review the settings. Here you can change the Display name to something more memorable. Click Create.
  12. Your instance is being created and will be accessible from the Services > Db2 Advanced tile. Also accessed from the Left menu My Instance then Click Provisioned instances
  13. From Provision Instances , you will see a list of instances. If you left the defaults, the name will be something like Db2 Advanced Edition-1. To get details Click on the instance name. In my case Db2 Advanced Edition-1. Note: You can see a red triangle that states Failed. This is temporary and is most likely the Cloud provision service waiting on a storage volume to come online and available to mount as a persistent volume to map the persistent volume claim.
  14. Click the horizontal 3 “.” on the right and see the options.
    • Open This will open the UI to allow you to work on the database
    • Manage Access This lets you control high level ACLs to the database.
    • Delete This will delete this particular instance.

    Back to Table of Contents

Create a Table

  1. When you Open the interface, you will run as the user id that provisioned it, for example user999 which is admin, so any tables that are created, by admin, via SQL editor by default will go under Schema user999.
    • You can use the UI and select and create tables under certain schemas.

Back to Table of Contents

Uninstalling DB2 Warehouse.

  1. First you will want to release any storage and delete instances to make sure storage is released. Use Delete to do this.
  2. From the command line:
  3. Set namespace. My namespace is zen your may be different like default
  4. Run env to verify that the following variables are exported
    • OpenShift 4.x 
         export NAMESPACE=zen
  5. Do a dry run uninstall to check what will be taken off. 
     ./cpd-cli uninstall --namespace ${NAMESPACE}  --assembly db2wh --uninstall-dry-run
  6. Run the uninstall 
     ./cpd-cli uninstall --namespace ${NAMESPACE}  --assembly db2wh
  7. Go to the Services catalog and verify that Db2 Warehouse is no longer enabled.

Back to Table of Contents

DataStage

Installing DataStage service

  1. Verify you have enough resource capcity to run DataStage. You many need to increase your work pool by a node.
  2. Run env to verify that the following variables are exported
    • OpenShift 4.x 
       export NAMESPACE=zen
 export STORAGE_CLASS=ibmc-file-gold-gid
 export DOCKER_REGISTRY_PREFIX=$(oc get routes image-registry -n openshift-image-registry -o template=\{\{.spec.host\}\})
 export LOCAL_REGISTRY=image-registry.openshift-image-registry.svc:5000
  3. Set the security aspects for DataStage to install properly 
      ./cpd-cli adm --repo ./repo.yaml  --namespace ${NAMESPACE} --apply --accept-all-licenses --assembly ds
  4. Deploy DataStage by running the following: 
      ./cpd-cli install --repo ./repo.yaml --namespace ${NAMESPACE} --storageclass ${STORAGE_CLASS} --transfer-image-to=${DOCKER_REGISTRY_PREFIX}/${NAMESPACE} --target-registry-username=ocadmin  --target-registry-password=$(oc whoami -t) --cluster-pull-prefix ${LOCAL_REGISTRY}/${NAMESPACE} --insecure-skip-tls-verify --latest-dependency --accept-all-licenses --assembly ds
  5. You will need to tab to accept the license.
  6. This will take some time to download, push to the registry, request new storage from IBM Cloud and provision the services and pods.
  7. Verify the installation 
      ./cpd-cli status --namespace ${NAMESPACE} --assembly ds
  8. Check for patches 
      ./cpd-cli status  --repo ./repo.yaml --namespace ${NAMESPACE} --patches --available-updates --assembly ds
  9. If there are patches apply the highest number as it will be cumulative. Some patches have prerequisite patches because they have dependencies on another service or on a set of shared, common services. If the patch details list one or more prerequisite patches, you must install the prerequisite patches before you install the service patch. You can run the following command to determine whether any of the prerequisite patches are already installed on the cluster:

Back to Table of Contents

Create a Transformation Project

  1. From the navigator click Organize > transform data. This will bring you into the Data Flow Designer in a projects view.
  2. If you do not see Organize > transform data, check your setting and profile in the upper right under the user icon. Click Profile and Setttings > Permissions then verify that you have following two permissions:
  3. Integrate and transform data
  4. View governance artifacts If not the users will need to have Data Engineer role.
    • Go to the Navigator on the left and scroll down to Administer Click on Manage Users.
    • On the right of the list of users, you will see a pencil, which indicates edit.
    • Click the pencil for the user you desire to edit.
    • Choose Data Engineer to get the appropriate permissions.
  5. From here you can create a new project.
    • Click + Create then provide a Project name.
    • Click Create button. This will take a minute or two.
  6. Click the Cog and there will be 4 options.
    • Adjust Compute Nodes //This allows you to scale nodes for a parallel job.
    • Configuration //Allows you to tweak service configurations
    • Multi-Cloud Kafka brokers (might be grayed out)
    • Server //Connections to Github and whether to use Machine Learning (On by default)
  7. Click on dstage1 to see how this works.
  8. Click on + Create, you should see a Parallel job and Sequence job
  9. Click ether job and a palette of nodes are available to you to build your job. These nodes are specific to the type of job.
  10. Across the top, you should see another cog. This allows you to add Smart Palatte to the options. This will start prompting you with potential options.
  11. Across the very top, is where you will add Connections, Table Definitions, Parameter Sets, Jobs and the current Job_1 that you were just creating.

Back to Table of Contents

Adding a new database driver

This is an example of adding a DB2 connector to DataStage on Cloud Pak for Data. You might try to a DB2 Connection and get a message Message id:RestApiResponseUnsuccessful 200 The DB2 library could not be loaded. This is suggesting that you need to load in or update a driver. Here are the instructions on how to accomplish this.

Back to Table of Contents

Adding a Netezza Driver

Netezza Client The Netezza Connector utilizes the Netezza ODBC drivers. These drivers need to be download and persisted on the Persistent Volume Claim (PVC)

  1. Install Steps The Netezza ODBC drivers can be downloaded from IBM Fix Central. Download the latest version of the drivers for use with the Netezza Connector.
  2. Download the Netezza drivers from Fix Central
  3. Copy the into the is-en-conductor-0 pod
    oc cp ./nps-linuxclient-v11.2.2.1.tar.gz zen/is-en-conductor-0:/mnt/dedicated_vol/Engine/is-en-conductor-0/EngineClients/
  4. Ensure you are in the context of the Engine/Conductor Pod: - oc exec -it is-en-conductor-0 bash - cd /mnt/dedicated_vol/Engine/is-en-conductor-0/EngineClients - Make a Directory for Netezza mkdir netezza - Copy or move the nps-linuxclient-v11.2.2.1.tar.gz into this directory.
    mv nps-linuxclient-v11.2.2.1.tar.gz netezza - Extract the tar ball tar xvzf nz-linuxclient-vXXXXX.tar.gz - Change directories into the extracted linux64 directory and run the installer. Make sure to change the directory where the NPS Linux Client will be unpacked to the PVC location
    • cd linux64
    • ./unpack
      Note: You will be prompted for a location. Please use /mnt/dedicated_vol/Engine/is-en-conductor-0/EngineClients/netezza/nz then use y to create the nz directory
        -------------------------------------------------------------------------------
  IBM IPS -- NPS Linux Client 11.2.2.1
  (C) Copyright IBM Corp. 2002, 2021  All Rights Reserved.
  -------------------------------------------------------------------------------
  Validating package checksum ... ok
  Where should the NPS Linux Client be unpacked? [/usr/local/nz] /mnt/dedicated_vol/Engine/is-en-conductor-0/EngineClients/netezza/nz
  Directory '/mnt/dedicated_vol/Engine/is-en-conductor-0/EngineClients/netezza/nz' does not exist; create it (y/n)? [y] y
  0%          25%         50%         75%          100%
  |||||||||||||||||||||||||||||||||||||||||||||||||||
  Unpacking complete.
    • The ODBC drivers can now be accessed at the following location: /mnt/dedicated_vol/Engine/is-en-conductor-0/EngineClients/netezza/nz/lib64
  5. Configuration Steps The next step is to configure the environment to use the new driver. First, create the ODBC DSN entry:
    1. Ensure you are still in the context of the Engine/Conductor Pod:
      oc exec -it is-en-conductor-0 bash
    2. Open the odbc.ini file vi /opt/IBM/InformationServer/Server/DSEngine/odbc.ini This launches you into a vi editor, make the following additions. Correct the Servername, Database, Username, Password field and another tweaks you might need to make.
        [ODBC Data Sources]
  # Existing section
  ...
  Netezza=Netezza Connection
  ...
  [Netezza]
  Driver=/mnt/dedicated_vol/Engine/is-en-conductor-0/EngineClients/netezza/nz/lib64/libnzodbc.so
  Description=NetezzaSQL ODBC
  Servername=NetezzaHost.ibm.com
  Port=5480
  Database=DatabaseName
  Username=
  Password=
  ReadOnly=false
  ShowSystemTables=false
  LegacySQLTables=false
  LoginTimeout=0
  QueryTimeout=0
  DateFormat=1
  NumericAsChar=false
  SQLBitOneZero=false
  StripCRLF=false
  securityLevel=preferredUnSecured
  DebugLogging=false
  6. Configure the environment within the dsenv:
    • Ensure you are still in the context of the Engine/Conductor Pod:
      oc exec -it is-en-conductor-0 bash
    • Open the dsenv file.
      vi /opt/IBM/InformationServer/Server/DSEngine/dsenv. This launches you into a vi editor, make the following additions 
      # Netezza
NETEZZA=/mnt/dedicated_vol/Engine/is-en-conductor-0/EngineClients/netezza/nz; export NETEZZA
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$NETEZZA/lib64; export LD_LIBRARY_PATH
NZ_ODBC_INI_PATH=$DSHOME; export NZ_ODBC_INI_PATH
  7. Restart the services:
    • This command will restart the Engine/Conductor pod
      oc delete po is-en-conductor-0 You are now ready to connect to Netezza using the Netezza Connector!

Back to Table of Contents

Using Legacy DataStage Designer

You can use the legacy DataStage Designer or Admin Client with Cloud Pak for Data. The current Transformation projects are not as robust ad the “thick” client. Here are the instructions on how to use the Legacy DataStage Designer.
The next gen implementation is currently in beta. on Cloud Pak for Data as a Service

  1. Create a project.
  2. Add to Project a DataStage Flow.

Back to Table of Contents

Quick Tutorial UI based Designer

While this tutorial is going against a Netezza database, it should also work with DB2 or others. Th first part of the setting up connections will be different and you will have a different database load. Here is a quick tutorial on using the UI based designer.

Back to Table of Contents

Run dsadmin commands

  1. You need to be in the is-en-conductor-0. It seems you need to exec in vs rsh or you get some library conflict issues.
  2. Log into OpenShift clusters
  3. Exec into is-en-conductor-0 oc exec -it is-en-conductor-0 bash
  4. Change directory to the DSEngine bin directory. cd /opt/IBM/InformationServer/Server/DSEngine/bin
  5. Run commands like ./dsadmin -listprojects 
      [dsadm@is-en-conductor-0 bin]$ ./dsadmin -listprojects
  ANALYZERPROJECT
  Datastage_Demo
  dstage1
  Mainline_DQ
  test_dt_project
  tomtest
  Status code = 0
  6. Enable Runtime Column Propagation: ./dsadmin -enablercp TRUE <project name> 
      [dsadm@is-en-conductor-0 bin]$ ./dsadmin -enablercp TRUE dstage1
  Status code = 0

Back to Table of Contents

Uninstalling DataStage.

  1. First you will want to release any storage and delete instances to make sure storage is released. Use Delete to do this.
  2. From the command line:
  3. Set namespace. My namespace is zen your may be different like default
  4. Run env to verify that the following variables are exported
    • OpenShift 4.x 
        export NAMESPACE=zen
  5. Do a dry run uninstall to check what will be taken off. 
       ./cpd-cli uninstall --namespace ${NAMESPACE}  --assembly ds --uninstall-dry-run
  6. Run the uninstall 
    ./cpd-cli uninstall --namespace ${NAMESPACE}  --assembly ds
  7. Go to the Services catalog and verify that DataStage is no longer enabled.

Back to Table of Contents

Installing Analytics Dashboards

Analytics Dashboards gives you a quick way to visualize your data. This is not full blown Cognos Analytics for Cloud Pak for Data* nor on premises version. Understand the current differences here

  1. Run env to verify that the following variables are exported
    • OpenShift 4.x 
       export NAMESPACE=zen
 export STORAGE_CLASS=ibmc-file-gold-gid
 export DOCKER_REGISTRY_PREFIX=$(oc get routes image-registry -n openshift-image-registry -o template=\{\{.spec.host\}\})
 export LOCAL_REGISTRY=image-registry.openshift-image-registry.svc:5000
  2. Set the security aspects for Analytics Dashboards to install properly 
      ./cpd-cli adm --repo ./repo.yaml  --namespace ${NAMESPACE} --apply --accept-all-licenses --assembly cde
  3. Deploy Analytics Dashboards by running the following: 
      ./cpd-cli install --repo ./repo.yaml --namespace ${NAMESPACE} --storageclass ${STORAGE_CLASS} --transfer-image-to=${DOCKER_REGISTRY_PREFIX}/${NAMESPACE} --target-registry-username=ocadmin  --target-registry-password=$(oc whoami -t) --cluster-pull-prefix ${LOCAL_REGISTRY}/${NAMESPACE} --insecure-skip-tls-verify --latest-dependency --accept-all-licenses --assembly cde
  4. You will need to tab to accept the license.
  5. This will take some time to download, push to the registry, request new storage from IBM Cloud and provision the services and pods.
  6. Verify the installation 
      ./cpd-cli status --namespace ${NAMESPACE} --assembly cde
  7. Check for patches 
      ./cpd-cli status  --repo ./repo.yaml --namespace ${NAMESPACE} --patches --available-updates --assembly cde
  8. If there are patches apply the highest number as it will be cumulative. Some patches have prerequisite patches because they have dependencies on another service or on a set of shared, common services. If the patch details list one or more prerequisite patches, you must install the prerequisite patches before you install the service patch. You can run the following command to determine whether any of the prerequisite patches are already installed on the cluster:

Back to Table of Contents

Uninstalling Analytics Dashboards

  1. From the command line:
  2. Set namespace. My namespace is zen your may be different like default
  3. Run env to verify that the following variables are exported
    • OpenShift 4.x 
        export NAMESPACE=zen
  4. Do a dry run uninstall to check what will be taken off. 
    ./cpd-cli uninstall --namespace ${NAMESPACE} --assembly cde --uninstall-dry-run
  5. Run the uninstall 
    ./cpd-cli uninstall --namespace ${NAMESPACE} --assembly cde
  6. Go to the Services catalog and verify that Analytics Dashboards is no longer enabled.

Back to Table of Contents

Installing Analytics Engine (Spark Clusters)

If you are using Data Refinery and you have to prep files larger than 100MB, then you will want to install the Analytics Engine to be able to select a different runtime than Data Refinery XS. This is the service to install if you want to use Spark with Watson Studio.

  1. Run env to verify that the following variables are exported
    • OpenShift 4.x 
       export NAMESPACE=zen
 export STORAGE_CLASS=ibmc-file-gold-gid
 export DOCKER_REGISTRY_PREFIX=$(oc get routes image-registry -n openshift-image-registry -o template=\{\{.spec.host\}\})
 export LOCAL_REGISTRY=image-registry.openshift-image-registry.svc:5000
  2. Set the security aspects for Analytics Engine to install properly 
      ./cpd-cli adm --repo ./repo.yaml  --namespace ${NAMESPACE} --apply --accept-all-licenses --assembly spark
  3. You will need an when installing the Spark assembly on a OCP 4.3 cluster. This is not needed for 3.11. Use this spark-ocp43-override.yaml
  4. Deploy Analytics Engine by running the following: 
      ./cpd-cli install --repo ./repo.yaml --namespace ${NAMESPACE} --storageclass ${STORAGE_CLASS} --transfer-image-to=${DOCKER_REGISTRY_PREFIX}/${NAMESPACE} --target-registry-username=ocadmin  --target-registry-password=$(oc whoami -t) --cluster-pull-prefix ${LOCAL_REGISTRY}/${NAMESPACE} --insecure-skip-tls-verify --latest-dependency --accept-all-licenses --assembly spark --override spark-ocp43-override.yaml
  5. You will need to tab to accept the license.
  6. This will take some time to download, push to the registry, request new storage from IBM Cloud and provision the services and pods.
  7. Verify the installation 
    ./cpd-cli status --namespace ${NAMESPACE} --assembly spark
  8. Check for patches 
      ./cpd-cli status  --repo ./repo.yaml --namespace ${NAMESPACE} --patches --available-updates --assembly spark
  9. If there are patches apply the highest number as it will be cumulative. Some patches have prerequisite patches because they have dependencies on another service or on a set of shared, common services. If the patch details list one or more prerequisite patches, you must install the prerequisite patches before you install the service patch. You can run the following command to determine whether any of the prerequisite patches are already installed on the cluster:

Back to Table of Contents

Provision Analytics Engine instance

coming soon Provision the instance

Back to Table of Contents

Resource constraints

coming soon Governing the about of resources in an instance

Uninstalling Analytics Engine

  1. From the command line:
  2. Set namespace. My namespace is zen your may be different like default
  3. Run env to verify that the following variables are exported
    • OpenShift 4.x 
         export NAMESPACE=zen
  4. Do a dry run uninstall to check what will be taken off. 
     ./cpd-cli uninstall --namespace ${NAMESPACE} --assembly spark --uninstall-dry-run
  5. Run the uninstall 
     ./cpd-cli uninstall --namespace ${NAMESPACE} --assembly spark
  6. Go to the Services catalog and verify that Analytics Engine is no longer enabled.

Back to Table of Contents

Cognos Analytics

Installing Cognos Analytics

Understand the current differences here

  1. Run env to verify that the following variables are exported.
    • OpenShift 4.x 
       export NAMESPACE=zen
 export STORAGE_CLASS=ibmc-file-gold-gid
 export DOCKER_REGISTRY_PREFIX=$(oc get routes image-registry -n openshift-image-registry -o template=\{\{.spec.host\}\})
 export LOCAL_REGISTRY=image-registry.openshift-image-registry.svc:5000
  2. Set the security aspects for Cognos to install properly 
    ./cpd-cli adm --repo ./repo.yaml  --namespace ${NAMESPACE} --apply --accept-all-licenses --assembly ca
  3. Deploy Cognos Analytics by running the following: 
    ./cpd-cli install --repo ./repo.yaml --namespace ${NAMESPACE} --storageclass ${STORAGE_CLASS} --transfer-image-to=${DOCKER_REGISTRY_PREFIX}/${NAMESPACE} --target-registry-username=ocadmin  --target-registry-password=$(oc whoami -t) --cluster-pull-prefix ${LOCAL_REGISTRY}/${NAMESPACE} --insecure-skip-tls-verify --latest-dependency --accept-all-licenses --assembly ca
  4. You will need to tab to accept the license.
  5. This will take some time to download, push to the registry, request new storage from IBM Cloud and provision the services and pods.
  6. Verify the installation 
      ./cpd-cli status --namespace ${NAMESPACE} --assembly ca
  7. Check for patches 
      ./cpd-cli status  --repo ./repo.yaml --namespace ${NAMESPACE} --patches --available-updates --assembly ca
  8. If there are patches apply the highest number as it will be cumulative. Some patches have prerequisite patches because they have dependencies on another service or on a set of shared, common services. If the patch details list one or more prerequisite patches, you must install the prerequisite patches before you install the service patch. You can run the following command to determine whether any of the prerequisite patches are already installed on the cluster:

Provision Cognos Analytics instance

  1. Preparing Db2 Advanced Database to make a connection that Cognos Analytics can used
  2. Collect the details of the provisioned database. This will be used for the environment variables and when creating the global connection. 
      Deployment id db2oltp-xxxxxxxxxxxxx
  Username userxxx
  Password xxxxxxxxxxxxxxxxxxxxxx
  JDBC URLjdbc:db2://HOST:PORT/BLUDB

    NOTE: If you do not run through the following steps to prepare the Database, your provision will fail.

  3. From a terminal window. Run the following to set up the environment to execute the Database updates. 
      export NAMESPACE=zen
  export DB2USERNAME=<USERNAME>
  export DB2PASSWORD=<PASSWORD>
  export DB2DEPLOYMENTID=<DEPLOYMENTID>
  4. Connect to the DB2 pods 
      export CMD="env DB2USERNAME=$DB2USERNAME DB2PASSWORD=$DB2PASSWORD  bash"
  oc  exec -ti c-$DB2DEPLOYMENTID-db2u-0 -n $NAMESPACE -- $CMD
  5. Run the following command to update the data base values from within the pod. 
      cd /mnt/blumeta0/home/db2inst1/sqllib
  . ./db2profile   

      db2 CONNECT to BLUDB user $DB2USERNAME using  $DB2PASSWORD;
  db2 UPDATE DATABASE CONFIGURATION USING APPLHEAPSZ 1024 DEFERRED;
  db2 UPDATE DATABASE CONFIGURATION USING LOCKTIMEOUT 240 DEFERRED;
  db2 CREATE BUFFERPOOL CMDB_08KBP IMMEDIATE SIZE 1000 PAGESIZE 8K;
  db2 CREATE BUFFERPOOL CMDB_32KBP IMMEDIATE SIZE 1000 PAGESIZE 32K;
  db2 CREATE SYSTEM TEMPORARY TABLESPACE TSN_SYS_CMDB IN DATABASE PARTITION GROUP IBMTEMPGROUP PAGESIZE 32K BUFFERPOOL CMDB_32KBP;
  db2 CREATE USER TEMPORARY TABLESPACE TSN_USR_CMDB IN DATABASE PARTITION GROUP IBMDEFAULTGROUP PAGESIZE 8K BUFFERPOOL CMDB_08KBP;
  db2 CREATE REGULAR TABLESPACE TSN_REG_CMDB IN DATABASE PARTITION GROUP IBMDEFAULTGROUP PAGESIZE 8K BUFFERPOOL CMDB_08KBP;
  db2 DROP TABLESPACE USERSPACE1;
  db2 CREATE SCHEMA db2COGNOS AUTHORIZATION $DB2USERNAME;
  db2 ALTER BUFFERPOOL ibmdefaultbp size 49800;
  6. Create a connection that will be used for Cognos Analytics.
  7. From the main menu with 3 horizontal bars, click on Connections.
  8. Provide a Name, I used Cognosdb
  9. Provide a Description (optional)
  10. Select a Connection Type, Select Db2. This will expand to give more options.
  11. You will focus on the following. You collected this at the top. - Host - This comes from the JDBC URL. It is the IP address - Port - This comes from the JDBC URL. It is the number after the IP: - Database - This comes from the JDBC URL. It is the number after the IP:PORT It is generally BLUDB - Username - Probably admin - Password - Same as collected above.
  12. Click the Test connection button to verify that it works properly.
  13. Click Create
  14. From the Services page, scroll to Cognos Analytic and click the tile
  15. in the upper right there is a button to Provision Instance

Back to Table of Contents

Accessing Watson Knowledge catalog

At some point, you will want to pull data from Watson Knowledge Catalog. There is a knowledge center page the tells you that it is possible, but never really tells you how to figure out the WKC JDBC URL and it is not obvious from the UI what this is. Digging around, you can create a connection from the Cognos UI.

  1. At the bottom of the main page on the left, you will find a button called Manage it has an icon with a person with a check mark. Click this Mange button.
  2. A slide out will happen to the right on the menu called Data Server Connections. Click the + symbol to the right of the search box.
  3. Another slide out will move right with Select Type at the top.
  4. Scroll through and select External Catalog
  5. Another slide out will reveal New data server connection and Edit External Catalog connection
  6. In the Edit External Catalog connections slide out, Enter jdbc:wkc:<route to CPD home minus /zen> in the Server url box.
    Example would look like. jdbc:wkc:https://zen-cpd-zen.testcluster.us-east.containers.appdomain.cloud
  7. Moving back left, leave the Authentication Method as Use the following signon:.
    • Click the +
    • This will slide out to let you enter username and password.
  8. Click the Test arrow to make sure you can connect to the Watson Knowledge Catalog.
  9. Assuming the connection came back successful, click Close.
  10. At the top, where it says New data server connection, click the little pencil to edit and name this external catalog. Example WKC Catalog.
  11. At the bottom click Save.
  12. Now go back and verify that you can get to the schemas.
    Note: Cognos Analytics will check all Catalogs that the user has access to. Then, only Connections with “Shared” credentials are read. If you have CSV files, or Connections w/ “private” credentials, these are not read by CA.

Back to Table of Contents

Uninstalling Cognos Analytics

  1. From the command line:
  2. Set namespace. My namespace is zen your may be different like default
  3. Run env to verify that the following variables are exported
    • OpenShift 4.x 
        export NAMESPACE=zen
  4. Do a dry run uninstall to check what will be taken off. 
      ./cpd-cli uninstall --namespace ${NAMESPACE} --assembly ca --uninstall-dry-run
  5. Run the uninstall 
      ./cpd-cli uninstall --namespace ${NAMESPACE} --assembly ca
  6. Go to the Services catalog and verify that Cognos Analytics is no longer enabled.

Back to Table of Contents

Watson Knowledge Catalog

Installing Watson Knowledge Catalog service

  1. Because of all the kernel changes, I generally suggest installing Watson Knowledge Catalog using the template during the control plane installation.
  2. Verify you have enough resource capacity to run Watson Knowledge Catalog. You many need to increase your work pool by a node.
  3. Create a secret in the kube-system project or namespace for the norootsquash.yaml will execute properly. Note: All norootsquash pods in kube-system namespace will restart using the new parameters.
    • oc project kube-system
    • Get the apikey from the repo.yaml file and use your accounts email address.
    • oc create secret docker-registry cpregistrysecret --docker-username=cp --docker-password= <API KEY used in repo.yaml > --docker-server=cp.icr.io/cp/cpd --docker-email= <your email address>
  4. Adjust the kernel parameters. Note: Save these 4 yaml files to the directory with cpd-cli command prior to executing or running the commands below.
  5. Run env to verify that the following variables are exported
    • OpenShift 4.x 
        export NAMESPACE=zen
  export STORAGE_CLASS=ibmc-file-gold-gid
  export DOCKER_REGISTRY_PREFIX=$(oc get routes image-registry -n openshift-image-registry -o template=\{\{.spec.host\}\})
  export LOCAL_REGISTRY=image-registry.openshift-image-registry.svc:5000
  6. Set the security aspects for Watson Knowledge Catalog to install properly 
    ./cpd-cli adm --repo ./repo.yaml  --namespace ${NAMESPACE} --apply --accept-all-licenses --assembly wkc
  7. Deploy Watson Knowledge Catalog by running the following: Download the override-nginx.yaml file for use in the install command. Note: You may desire to change the timezone in the override-nginx.yaml file to accommodate your timezone. Examples: America/Chicago, Europe/London, Asia/Tapei 
    ./cpd-cli install --repo ./repo.yaml --namespace ${NAMESPACE} --storageclass ${STORAGE_CLASS} --transfer-image-to=${DOCKER_REGISTRY_PREFIX}/${NAMESPACE} --target-registry-username=ocadmin  --target-registry-password=$(oc whoami -t) --cluster-pull-prefix ${LOCAL_REGISTRY}/${NAMESPACE} --insecure-skip-tls-verify --latest-dependency --accept-all-licenses --override override-nginx.yaml --assembly wkc
  8. You will need to tab to accept the license.
  9. This will take some time to download, push to the registry, request new storage from IBM Cloud and provision the services and pods.
  10. Verify the installation 
     ./cpd-cli status --namespace ${NAMESPACE} --assembly wkc
  11. Check for patches 
     ./cpd-cli status  --repo ./repo.yaml --namespace ${NAMESPACE} --patches --available-updates --assembly wkc
  12. If there are patches apply the highest number as it will be cumulative. Some patches have prerequisite patches because they have dependencies on another service or on a set of shared, common services. If the patch details list one or more prerequisite patches, you must install the prerequisite patches before you install the service patch. You can run the following command to determine whether any of the prerequisite patches are already installed on the cluster:

Back to Table of Contents

Post install tasks (To Do’s for Tom to validate)

  1. Setting up your default catalog
  2. Optional tasks

Uninstalling Watson Knowledge Catalog

  1. From the command line:
  2. Set namespace. My namespace is zen your may be different like default
  3. Run env to verify that the following variables are exported
    • OpenShift 4.x 
      export NAMESPACE=zen
  4. Do a dry run uninstall to check what will be taken off. 
    ./cpd-cli uninstall --namespace ${NAMESPACE} --assembly wkc --uninstall-dry-run
  5. Run the uninstall 
    ./cpd-cli uninstall --namespace ${NAMESPACE} --assembly wkc

Back to Table of Contents

Watson Discovery

Installing Watson Discovery service

  1. There are a few things that you need to do in an override file to get this to install properly. Here is your starting point wd-override.yaml
  2. Edit this wd-override.yaml file. Be careful of spaces, yaml is touchy.
    • deploymentType: "Development" This should stay Development but if you wanted Production here is where you would change this. This adds scaling to the deployment.
    • Changing if Content Intelligence is enabled in the file. If you want this then change this part to true 
      wdRelease:
 deploymentType: Production
 enableContentIntelligence: true
  3. Run env to verify that the following variables are exported
    • OpenShift 4.x 
         export NAMESPACE=zen
   export STORAGE_CLASS=ibmc-block-gold
   export DOCKER_REGISTRY_PREFIX=$(oc get routes image-registry -n openshift-image-registry -o template=\{\{.spec.host\}\})
   export LOCAL_REGISTRY=image-registry.openshift-image-registry.svc:5000
  4. Update the repo file to look like this repo.watson-discovery.yaml. Update the apikey to match yours. Note: for some reason in 3.5.2 cli the namespaces are not recognized, so I added to the url. However the modules look for the registry name, so there needs to be the extra entries.
  5. Set the security aspects for Watson Discovery to install properly 
    ./cpd-cli adm --repo ./repo.watson-discovery.yaml  --namespace ${NAMESPACE} --apply --accept-all-licenses --assembly watson-discovery
  6. Deploy Watson Discovery happens in 2 steps by running the following: Download the wd-override.yaml file for use in the install command
  7. First you will install the edb-operator: 
     ./cpd-cli install --repo ./repo.watson-discovery.yaml --namespace ${NAMESPACE} --storageclass ${STORAGE_CLASS} --transfer-image-to=${DOCKER_REGISTRY_PREFIX}/${NAMESPACE} --target-registry-username=ocadmin  --target-registry-password=$(oc whoami -t) --cluster-pull-prefix ${LOCAL_REGISTRY}/${NAMESPACE} --insecure-skip-tls-verify --latest-dependency --accept-all-licenses  --assembly edb-operator --optional-modules edb-pg-base:x86_64 --override ./wd-override.yaml
    • You will need to tab to accept the license.
    • This will take some time to download, push to the registry, request new storage from IBM Cloud and provision the services and pods.
  8. Second install the the watson-discovery assembly 
     ./cpd-cli install --repo ./repo.watson-discovery.yaml --namespace ${NAMESPACE} --storageclass ${STORAGE_CLASS} --transfer-image-to=${DOCKER_REGISTRY_PREFIX}/${NAMESPACE} --target-registry-username=ocadmin  --target-registry-password=$(oc whoami -t) --cluster-pull-prefix ${LOCAL_REGISTRY}/${NAMESPACE} --insecure-skip-tls-verify --latest-dependency --accept-all-licenses --assembly watson-discovery --override ./wd-override.yaml

    - This will take some time to download, push to the registry, request new storage from IBM Cloud and provision the services and pods.
  9. Verify the installation 
      ./cpd-cli status --namespace ${NAMESPACE} --assembly watson-discovery
  10. Check for patches 
      ./cpd-cli status  --repo ./repo.yaml --namespace ${NAMESPACE} --patches --available-updates --assembly watson-discovery
  11. If there are patches apply the highest number as it will be cumulative. Some patches have prerequisite patches because they have dependencies on another service or on a set of shared, common services. If the patch details list one or more prerequisite patches, you must install the prerequisite patches before you install the service patch. You can run the following command to determine whether any of the prerequisite patches are already installed on the cluster:

    Back to Table of Contents

Post install tasks (To Do’s for Tom to validate)

  1. TBD

Back to Table of Contents

Uninstalling Watson Discovery

  1. From the command line:
  2. Set namespace. My namespace is zen your may be different like default
  3. Run env to verify that the following variables are exported
    • OpenShift 4.x 
      export NAMESPACE=zen
  4. Do a dry run uninstall to check what will be taken off. 
    ./cpd-cli uninstall --namespace ${NAMESPACE} --include-dependent-assemblies --assembly watson-discovery --uninstall-dry-run
  5. Run the uninstall 
    ./cpd-cli uninstall --namespace ${NAMESPACE} --include-dependent-assemblies --assembly watson-discovery
  6. Post uninstall clean up tasks.
    • Look for existing Persistent Volume Claims (PVC) that were not deleted by running oc get pvc | grep "wd-" 
      tjm$ oc get pvc | grep "wd-"
data-wd-rabbitmq-discovery-0                                        Bound    pvc-82769061-0908-41dd-93d5-38b4dc0a3e31   20Gi       RWO            ibmc-block-gold             11h
export-wd-minio-discovery-0                                         Bound    pvc-ca94ea59-4364-460d-a938-5408417dd5ce   100Gi      RWO            ibmc-block-gold             11h
export-wd-minio-discovery-1                                         Bound    pvc-c61c829d-c8bc-4186-95cc-fa1bf7ca4e68   100Gi      RWO            ibmc-block-gold             11h
export-wd-minio-discovery-2                                         Bound    pvc-fd4b9f1c-255c-4e49-aeb9-0c7b88be5db9   100Gi      RWO            ibmc-block-gold             11h
export-wd-minio-discovery-3                                         Bound    pvc-3f3e3dc9-e98e-44a1-bb24-7ecef52dc316   100Gi      RWO            ibmc-block-gold             11h
wd-ibm-elasticsearch-data-wd-ibm-elasticsearch-es-server-client-0   Bound    pvc-d9519e68-a63f-4237-afaf-ed730cb535ff   20Gi       RWO            ibmc-block-gold             11h
wd-ibm-elasticsearch-data-wd-ibm-elasticsearch-es-server-data-0     Bound    pvc-706347a9-aa28-47f9-a914-5eb445c12694   40Gi       RWO            ibmc-block-gold             11h
wd-ibm-elasticsearch-data-wd-ibm-elasticsearch-es-server-master-0   Bound    pvc-3d790b36-0c4f-4ef1-a926-2f208635fd54   20Gi       RWO            ibmc-block-gold             11h
  7. Delete these pvcs oc delete pvc <pvc name> <pvc name> 
    tjm$ oc delete pvc data-wd-rabbitmq-discovery-0   export-wd-minio-discovery-0    export-wd-minio-discovery-1   export-wd-minio-discovery-2  export-wd-minio-discovery-3  wd-ibm-elasticsearch-data-wd-ibm-elasticsearch-es-server-client-0 wd-ibm-elasticsearch-data-wd-ibm-elasticsearch-es-server-data-0 wd-ibm-elasticsearch-data-wd-ibm-elasticsearch-es-server-master-0  
persistentvolumeclaim "data-wd-rabbitmq-discovery-0" deleted
persistentvolumeclaim "export-wd-minio-discovery-0" deleted
persistentvolumeclaim "export-wd-minio-discovery-1" deleted
persistentvolumeclaim "export-wd-minio-discovery-2" deleted
persistentvolumeclaim "export-wd-minio-discovery-3" deleted
persistentvolumeclaim "wd-ibm-elasticsearch-data-wd-ibm-elasticsearch-es-server-client-0" deleted
persistentvolumeclaim "wd-ibm-elasticsearch-data-wd-ibm-elasticsearch-es-server-data-0" deleted
persistentvolumeclaim "wd-ibm-elasticsearch-data-wd-ibm-elasticsearch-es-server-master-0" deleted
  8. You may need to patch the Finalizer state to complete the delete. Command is as follows kubectl patch pvc <pvc name> -p '{"metadata":{"finalizers": []}}' --type=merge
  9. Check that they are all gone now. 
    tjm $ oc get pvc | grep "wd-"
tjm $

Back to Table of Contents

Watson Knowledge Studio

Installing Watson Knowledge Studio service

  1. There are a few things that you need to do in an override file to get this to install properly. Here is your starting point wks-override.yaml
  2. Edit this wks-override.yaml file. Be careful of spaces, yaml is touchy.
    - The AWT service needs a RWX StorageClass. Block is RWO. 
    wawt:
  persistentVolume:
    storageClassName: ibmc-file-custom-gold-gid
  3. Run env to verify that the following variables are exported - OpenShift 4.x ~~~ export NAMESPACE=zen export STORAGE_CLASS=ibmc-block-gold export DOCKER_REGISTRY_PREFIX=$(oc get routes image-registry -n openshift-image-registry -o template={{.spec.host}}) export LOCAL_REGISTRY=image-registry.openshift-image-registry.svc:5000 ~~~
  4. Update the repo file to look like this repo.wks.yaml. Update the apikey to match yours.
  5. Collect existing configuration prior to getting delta images. This produces a config.yaml file that you will include in the next step. 
    ./cpd-cli collect-config --namespace ${NAMESPACE
[INFO] [2021-05-11 15:00:39-0179] Deployment specification saved at config.yaml
  6. Download certain set of containers that seem to not install with the main install. 
    ./cpd-cli delta-images --assembly-list ./download.yaml  -r ./repo.wks.yaml --action download --accept-all-licenses --config-yaml config.yaml --download-path ./cpd-cli-workspace/cpdfiles
  7. Push the downloaded image to the registry. 
    ./cpd-cli delta-images --assembly-list ./download.yaml --transfer-image-to ${DOCKER_REGISTRY_PREFIX}/${NAMESPACE}   --target-registry-password $(oc whoami -t) --target-registry-username ocadmin --insecure-skip-tls-verify --action push --load-from ./cpd-cli-workspace/cpdfiles  --accept-all-licenses
  8. Set the security aspects for Watson Discovery to install properly 
    ./cpd-cli adm --repo ./repo.wks.yaml  --namespace ${NAMESPACE} --apply --accept-all-licenses --assembly watson-ks
  9. Deploy Watson Knowledge studio needs EDB installed and a different storage class which is included in the override file: Download the wks-override.yaml file for use in the install command 1. Install the the watson-ks assembly 
    ./cpd-cli install --repo ./repo.wks.yaml --namespace ${NAMESPACE} --storageclass ${STORAGE_CLASS} --transfer-image-to=${DOCKER_REGISTRY_PREFIX}/${NAMESPACE} --target-registry-username=ocadmin  --target-registry-password=$(oc whoami -t) --cluster-pull-prefix ${LOCAL_REGISTRY}/${NAMESPACE} --insecure-skip-tls-verify --latest-dependency --accept-all-licenses  --assembly watson-ks --optional-modules edb-pg-base:x86_64 --override ./wks-override.yaml
    • This will take some time to download, push to the registry, request new storage from IBM Cloud and provision the services and pods.
    • If something goes wrong, you can use the following to patch. Here is the wks-spec.yaml file 
      oc patch wks wks --type=merge --patch="$(cat wks-spec.yaml)"
  10. Verify the installation 
      ./cpd-cli status --namespace ${NAMESPACE} --assembly watson-ks
  11. Check for patches 
      ./cpd-cli status  --repo ./repo.yaml --namespace ${NAMESPACE} --patches --available-updates --assembly watson-ksß
  12. If there are patches apply the highest number as it will be cumulative. Some patches have prerequisite patches because they have dependencies on another service or on a set of shared, common services. If the patch details list one or more prerequisite patches, you must install the prerequisite patches before you install the service patch. You can run the following command to determine whether any of the prerequisite patches are already installed on the cluster:

    Back to Table of Contents

Post install tasks (To Do’s for Tom to validate)

  1. TBD

Back to Table of Contents

Uninstalling Watson Knowledge Studio

  1. From the command line:
  2. Set namespace. My namespace is zen your may be different like default
  3. Run env to verify that the following variables are exported - OpenShift 4.x ~~~ export NAMESPACE=zen
    ~~~
  4. Do a dry run uninstall to check what will be taken off. 
    ./cpd-cli uninstall --namespace ${NAMESPACE} --include-dependent-assemblies --assembly watson-ks --uninstall-dry-run
  5. Run the uninstall 
    ./cpd-cli uninstall --namespace ${NAMESPACE} --include-dependent-assemblies --assembly watson-ks
  6. Check to see if any artifacts are left over. 
    oc -n ${NAMESPACE} get deploy,sts,svc,ing,sa,role,rolebinding,secret,cm,netpol,hpa,job,pvc,pod -l "release in (wks, wks-minio, wks-ibm-watson-ks)" --ignore-not-found
  7. If there are then you can clean them up. ~~~ oc -n ${NAMESPACE} delete deploy,sts,svc,ing,sa,role,rolebinding,secret,cm,netpol,hpa,job,pvc,pod -l “release in (wks, wks-minio, wks-ibm-watson-ks)” –ignore-not-found ~~~

Back to Table of Contents