OBIEE 11g HA Implementation


Introduction 

We have successfully implemented Active/Passive Topology design for OBIEE 11g application. 

Software/OS version used: 

OBIEE 11g 111.1.1.5.0 (single instance database 11.2.0.3.0) running on top of Oracle 11g 11.2.0.3.0 CRS (aka integrated with GNS/DHCP/dynamic SCAN configuration). Linux : RHEL 5.6 and SAN storage used. 

Technical briefing: 

1. Used 2 shared storage devices : 1 for ASM(Vote/OCR) and 1 for OBIEE fusion middleware

2. 2 Virtual IP used – 1 for database listener and 1 for OBIEE  application.

3. 2 hosts used – in first host (say OMIPRI) database will be actively running and OBIEE will be in standby mode. In 2nd host(OMISEC), OBIEE will be active and Database will be in standby mode.

So if  OMIPRI crashes, Database will move to OIMSEC or if OMISEC crashes, OBIEE will migrate to OMIPRI. So in net , at any point in time, OBIEE application will be active.

All the 3 (A,B and C) managed via CRS.

4.OBIEEVIP, Database VIP, GNS, SCAN addresses resolved via DNS. This configuration supports both HA and scaling. When both nodes up and running, HA is 100% and it drops to 0% HA when there is only one node  is active at any point in time.

 

Image

 

When both nodes are active: Database running on OIMPRI and OBIEE running on OIMSEC

———————————————————————————-

 

[root@oimpri ~]# ps -ef|grep tns

oracle    5707     1  0 09:31 ?        00:00:00 /dbfiles/app/oracle/product/11.2.0/db/bin/tnslsnr LISTENER -inherit

oracle    6047     1  0 09:32 ?        00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN2 -inherit

oracle    6076     1  0 09:32 ?        00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN3 -inherit

root      9999 26162  0 11:22 pts/4    00:00:00 grep tns

[root@oimpri ~]# ps -ef|grep pmon

oracle    4892     1  0 09:29 ?        00:00:00 asm_pmon_+ASM1

oracle    5987     1  0 09:32 ?        00:00:00 ora_pmon_oimap

root     10001 26162  0 11:22 pts/4    00:00:00 grep pmon

[root@oimpri ~]#

 

Name                         Type                Target     State

———————————————————————————–

ora.scan2.vip                Other               ONLINE     ONLINE on oimpri

ora.oimsec.vip               Other               ONLINE     ONLINE on oimsec

ora.LISTENER_SCAN3.lsnr      Other               ONLINE     ONLINE on oimpri

ora.oc4j                     Other               ONLINE     ONLINE on oimpri

ora.gsd                      Other               OFFLINE    OFFLINE

ora.asdb.cfcdb               Database Instance   ONLINE     OFFLINE

ora.oimsec.ASM2.asm          Other               ONLINE     ONLINE on oimsec

ora.asdblsnr.cfcdblsnr       Database Listener   ONLINE     ONLINE on oimpri

ora.obieevip.cfcvip          Virtual IP          ONLINE     ONLINE on oimsec

ora.LISTENER_SCAN2.lsnr      Other               ONLINE     ONLINE on oimpri

ora.obieeserver1.cfcas       Opmn Instance       ONLINE     ONLINE on oimsec

ora.oimap.db                 Other               ONLINE     ONLINE on oimpri

ora.obieeserver.cfcas        WebLogic Domain     ONLINE     ONLINE on oimsec

ora.oimsec.gsd               Other               OFFLINE    OFFLINE

ora.LISTENER_SCAN1.lsnr      Other               ONLINE     ONLINE on oimsec

ora.dbvip.cfcvip             Virtual IP          ONLINE     ONLINE on oimpri

ora.gns.vip                  Other               ONLINE     ONLINE on oimpri

ora.oimpri.ons               Other               ONLINE     ONLINE on oimpri

ora.asm                      Other               ONLINE     ONLINE on oimpri

ora.gns                      Other               ONLINE     ONLINE on oimpri

ora.DATA.dg                  Other               ONLINE     ONLINE on oimpri

ora.net1.network             Other               ONLINE     ONLINE on oimpri

ora.cvu                      Other               ONLINE     ONLINE on oimpri

ora.oimsec.ons               Other               ONLINE     OFFLINE

ora.scan3.vip                Other               ONLINE     ONLINE on oimpri

ora.scan1.vip                Other               ONLINE     ONLINE on oimsec

ora.dbdisk.cfcdisk           Shared Disk         ONLINE     ONLINE on oimpri

ora.registry.acfs            Other               ONLINE     ONLINE on oimpri

ora.ons                      Other               ONLINE     ONLINE on oimpri

ora.obieedisk.cfcdisk        Shared Disk         ONLINE     ONLINE on oimsec

ora.oimpri.ASM1.asm          Other               ONLINE     ONLINE on oimpri

ora.oimpri.vip               Other               ONLINE     ONLINE on oimpri

ora.oimpri.gsd               Other               OFFLINE    OFFLINE

 

  1* select username,machine,count(*) from v$session where username is not null group by username,machine

SQL> /

 

USERNAME                       MACHINE                                                            COUNT(*)

—————————— —————————————————————- ———-

SYS                            oimpri                                                                    5

DEV_MDS                        oimsec                                                                    16

 

SQL>

Image

After the Database node (OIMPRI) crash: Database switched to OIMSEC. Now both database and OBIEE running on the same node: OIMSEC

————————————————————————————————-

 

 

On OIMSEC Node After the Database Failover:

 

 

[oracle@oimsec bin]$ ps -ef|grep  tns

oracle     775     1  0 08:13 ?        00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN1 -inherit

oracle    1058     1  0 08:13 ?        00:00:00 /dbfiles/app/oracle/product/11.2.0/db/bin/tnslsnr LISTENER -inherit

oracle   16112     1  0 07:45 ?        00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN3 -inherit

oracle   16121     1  0 07:45 ?        00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN2 -inherit

 

[oracle@oimsec bin]$ ps -ef|grep pmon

oracle    1279     1  0 08:13 ?        00:00:00 ora_pmon_oimap

oracle    4742     1  0 07:04 ?        00:00:00 asm_pmon_+ASM2

 

Name                         Type                Target     State

———————————————————————————–

ora.scan2.vip                Other               ONLINE     ONLINE on oimsec

ora.oimsec.vip               Other               ONLINE     ONLINE on oimsec

ora.LISTENER_SCAN3.lsnr      Other               ONLINE     ONLINE on oimsec

ora.oc4j                     Other               ONLINE     ONLINE on oimsec

ora.gsd                      Other               OFFLINE    OFFLINE

ora.asdb.cfcdb               Database Instance   ONLINE     ONLINE on oimsec

ora.oimsec.ASM2.asm          Other               ONLINE     ONLINE on oimsec

ora.asdblsnr.cfcdblsnr       Database Listener   ONLINE     ONLINE on oimsec

ora.obieevip.cfcvip          Virtual IP          ONLINE     ONLINE on oimsec

ora.LISTENER_SCAN2.lsnr      Other               ONLINE     ONLINE on oimsec

ora.obieeserver1.cfcas       Opmn Instance       ONLINE     ONLINE on oimsec

ora.oimap.db                 Other               ONLINE     OFFLINE

ora.obieeserver.cfcas        WebLogic Domain     ONLINE     ONLINE on oimsec

ora.oimsec.gsd               Other               OFFLINE    OFFLINE

ora.LISTENER_SCAN1.lsnr      Other               ONLINE     ONLINE on oimsec

ora.dbvip.cfcvip             Virtual IP          ONLINE     ONLINE on oimsec

ora.gns.vip                  Other               ONLINE     ONLINE on oimsec

ora.asm                      Other               ONLINE     ONLINE on oimsec

ora.gns                      Other               ONLINE     ONLINE on oimsec

ora.DATA.dg                  Other               ONLINE     ONLINE on oimsec

ora.net1.network             Other               ONLINE     ONLINE on oimsec

ora.cvu                      Other               ONLINE     ONLINE on oimsec

ora.oimsec.ons               Other               ONLINE     ONLINE on oimsec

ora.scan3.vip                Other               ONLINE     ONLINE on oimsec

ora.scan1.vip                Other               ONLINE     ONLINE on oimsec

ora.dbdisk.cfcdisk           Shared Disk         ONLINE     ONLINE on oimsec

ora.registry.acfs            Other               ONLINE     ONLINE on oimsec

ora.ons                      Other               ONLINE     ONLINE on oimsec

ora.obieedisk.cfcdisk        Shared Disk         ONLINE     ONLINE on oimsec

ora.oimpri.vip               Other               ONLINE     ONLINE on oimsec

 

 1* select username,machine,count(*) from v$session where username is not null group by username,machine

SQL> /

 

USERNAME                       MACHINE                                                            COUNT(*)

—————————— —————————————————————- ———-

SYS                                     oimpri                                                                    5

DEV_MDS                        oimsec                                                                    4

 

 Image

After the OBIEE node (OIMSEC) crash: OBIEE and Database switched to OIMPRI. OMIPRI is already up without any servicing running.

————————————————————————————————-

 

On oimpri Node After the Database Failover:

 

 

[oracle@oimpri bin]$ ps -ef|grep  tns

oracle     775     1  0 08:13 ?        00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN1 -inherit

oracle    1058     1  0 08:13 ?        00:00:00 /dbfiles/app/oracle/product/11.2.0/db/bin/tnslsnr LISTENER -inherit

oracle   16112     1  0 07:45 ?        00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN3 -inherit

oracle   16121     1  0 07:45 ?        00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN2 -inherit

 

[oracle@oimpri bin]$ ps -ef|grep pmon

oracle    1279     1  0 08:13 ?        00:00:00 ora_pmon_oimap

oracle    4742     1  0 07:04 ?        00:00:00 asm_pmon_+ASM2

 

Name                         Type                Target     State

———————————————————————————–

ora.scan2.vip                Other               ONLINE     ONLINE on oimpri

ora.oimpri.vip               Other               ONLINE     ONLINE on oimpri

ora.LISTENER_SCAN3.lsnr      Other               ONLINE     ONLINE on oimpri

ora.oc4j                     Other               ONLINE     ONLINE on oimpri

ora.gsd                      Other               OFFLINE    OFFLINE

ora.asdb.cfcdb               Database Instance   ONLINE     ONLINE on oimpri

ora.oimpri.ASM2.asm          Other               ONLINE     ONLINE on oimpri

ora.asdblsnr.cfcdblsnr       Database Listener   ONLINE     ONLINE on oimpri

ora.obieevip.cfcvip          Virtual IP          ONLINE     ONLINE on oimpri

ora.LISTENER_SCAN2.lsnr      Other               ONLINE     ONLINE on oimpri

ora.obieeserver1.cfcas       Opmn Instance       ONLINE     ONLINE on oimpri

ora.oimap.db                 Other               ONLINE     OFFLINE

ora.obieeserver.cfcas        WebLogic Domain     ONLINE     ONLINE on oimpri

ora.oimpri.gsd               Other               OFFLINE    OFFLINE

ora.LISTENER_SCAN1.lsnr      Other               ONLINE     ONLINE on oimpri

ora.dbvip.cfcvip             Virtual IP          ONLINE     ONLINE on oimpri

ora.gns.vip                  Other               ONLINE     ONLINE on oimpri

ora.asm                      Other               ONLINE     ONLINE on oimpri

ora.gns                      Other               ONLINE     ONLINE on oimpri

ora.DATA.dg                  Other               ONLINE     ONLINE on oimpri

ora.net1.network             Other               ONLINE     ONLINE on oimpri

ora.cvu                      Other               ONLINE     ONLINE on oimpri

ora.oimpri.ons               Other               ONLINE     ONLINE on oimpri

ora.scan3.vip                Other               ONLINE     ONLINE on oimpri

ora.scan1.vip                Other               ONLINE     ONLINE on oimpri

ora.dbdisk.cfcdisk           Shared Disk         ONLINE     ONLINE on oimpri

ora.registry.acfs            Other               ONLINE     ONLINE on oimpri

ora.ons                      Other               ONLINE     ONLINE on oimpri

ora.obieedisk.cfcdisk        Shared Disk         ONLINE     ONLINE on oimpri

ora.oimpri.vip               Other               ONLINE     ONLINE on oimpri

 

 1* select username,machine,count(*) from v$session where username is not null group by username,machine

SQL> /

 USERNAME                       MACHINE                                                            COUNT(*)

—————————— —————————————————————- ———-

SYS                                   oimpri                                                                    5

DEV_MDS                        oimpri                                                                    7

 Maintenance o OMISEC is completed, so that OBIEE resources manually switched to OIMSEC

—————————————————————————————

 [root@oimpri ~]# ps -ef|grep tns

oracle    5707     1  0 09:31 ?        00:00:00 /dbfiles/app/oracle/product/11.2.0/db/bin/tnslsnr LISTENER -inherit

oracle    6047     1  0 09:32 ?        00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN2 -inherit

oracle    6076     1  0 09:32 ?        00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN3 -inherit

root      9999 26162  0 11:22 pts/4    00:00:00 grep tns

[root@oimpri ~]# ps -ef|grep pmon

oracle    4892     1  0 09:29 ?        00:00:00 asm_pmon_+ASM1

oracle    5987     1  0 09:32 ?        00:00:00 ora_pmon_oimap

root     10001 26162  0 11:22 pts/4    00:00:00 grep pmon

[root@oimpri ~]#

 

Name                         Type                Target     State

———————————————————————————–

ora.scan2.vip                Other               ONLINE     ONLINE on oimpri

ora.oimsec.vip               Other               ONLINE     ONLINE on oimsec

ora.LISTENER_SCAN3.lsnr      Other               ONLINE     ONLINE on oimpri

ora.oc4j                     Other               ONLINE     ONLINE on oimpri

ora.gsd                      Other               OFFLINE    OFFLINE

ora.asdb.cfcdb               Database Instance   ONLINE     OFFLINE

ora.oimsec.ASM2.asm          Other               ONLINE     ONLINE on oimsec

ora.asdblsnr.cfcdblsnr       Database Listener   ONLINE     ONLINE on oimpri

ora.obieevip.cfcvip          Virtual IP          ONLINE     ONLINE on oimsec

ora.LISTENER_SCAN2.lsnr      Other               ONLINE     ONLINE on oimpri

ora.obieeserver1.cfcas       Opmn Instance       ONLINE     ONLINE on oimsec

ora.oimap.db                 Other               ONLINE     ONLINE on oimpri

ora.obieeserver.cfcas        WebLogic Domain     ONLINE     ONLINE on oimsec

ora.oimsec.gsd               Other               OFFLINE    OFFLINE

ora.LISTENER_SCAN1.lsnr      Other               ONLINE     ONLINE on oimsec

ora.dbvip.cfcvip             Virtual IP          ONLINE     ONLINE on oimpri

ora.gns.vip                  Other               ONLINE     ONLINE on oimpri

ora.oimpri.ons               Other               ONLINE     ONLINE on oimpri

ora.asm                      Other               ONLINE     ONLINE on oimpri

ora.gns                      Other               ONLINE     ONLINE on oimpri

ora.DATA.dg                  Other               ONLINE     ONLINE on oimpri

ora.net1.network             Other               ONLINE     ONLINE on oimpri

ora.cvu                      Other               ONLINE     ONLINE on oimpri

ora.oimsec.ons               Other               ONLINE     OFFLINE

ora.scan3.vip                Other               ONLINE     ONLINE on oimpri

ora.scan1.vip                Other               ONLINE     ONLINE on oimsec

ora.dbdisk.cfcdisk           Shared Disk         ONLINE     ONLINE on oimpri

ora.registry.acfs            Other               ONLINE     ONLINE on oimpri

ora.ons                      Other               ONLINE     ONLINE on oimpri

ora.obieedisk.cfcdisk        Shared Disk         ONLINE     ONLINE on oimsec

ora.oimpri.ASM1.asm          Other               ONLINE     ONLINE on oimpri

ora.oimpri.vip               Other               ONLINE     ONLINE on oimpri

ora.oimpri.gsd               Other               OFFLINE    OFFLINE

 

  1* select username,machine,count(*) from v$session where username is not null group by username,machine

SQL> /

 USERNAME                       MACHINE                                                            COUNT(*)

—————————— —————————————————————- ———-

SYS                                    oimpri                                                                    5

DEV_MDS                        oimsec                                                                    3

 

 

 

 

All resource dependency trees

————————————————-

 ec.vip

  ora.LISTENER_SCAN3.lsnr

  ora.oc4j

  ora.gsd

  ora.asdb.cfcdb

    |

    +->ora.asdblsnr.cfcdblsnr

    |    |

    |    +->ora.dbvip.cfcvip

    |    |

    |    +->ora.dbdisk.cfcdisk

    |

    +->ora.dbdisk.cfcdisk

  ora.oimsec.ASM2.asm

  ora.LISTENER_SCAN2.lsnr

  ora.obieeserver1.cfcas

    |

    +->ora.obieevip.cfcvip

    |

    +->ora.obieedisk.cfcdisk

  ora.oimap.db

  ora.obieeserver.cfcas

    |

    +->ora.obieevip.cfcvip

    |

    +->ora.obieedisk.cfcdisk

  ora.oimsec.gsd

  ora.LISTENER_SCAN1.lsnr

  ora.gns.vip

  ora.asm

  ora.gns

  ora.DATA.dg

  ora.net1.network

  ora.cvu

  ora.oimsec.ons

  ora.scan3.vip

  ora.scan1.vip

  ora.registry.acfs

  ora.ons

  ora.oimpri.vip

 

 

Cluster Resources

——————————————————————————–

ora.LISTENER_SCAN1.lsnr

      1        ONLINE  ONLINE       oimpri

ora.LISTENER_SCAN2.lsnr

      1        ONLINE  ONLINE       oimsec

ora.LISTENER_SCAN3.lsnr

      1        ONLINE  ONLINE       oimsec

ora.asdb.cfcdb

      1        ONLINE  ONLINE       oimsec

ora.asdblsnr.cfcdblsnr

      1        ONLINE  ONLINE       oimsec

ora.cvu

      1        ONLINE  ONLINE       oimsec

ora.dbdisk.cfcdisk

      1        ONLINE  ONLINE       oimsec

ora.dbvip.cfcvip

      1        ONLINE  ONLINE       oimsec

ora.gns

      1        ONLINE  ONLINE       oimsec

ora.gns.vip

      1        ONLINE  ONLINE       oimsec

ora.obieedisk.cfcdisk

      1        ONLINE  ONLINE       oimsec

ora.obieeserver.cfcas

      1        ONLINE  ONLINE       oimsec

ora.obieeserver1.cfcas

      1        ONLINE  OFFLINE

ora.obieevip.cfcvip

      1        ONLINE  ONLINE       oimsec

ora.oc4j

      1        ONLINE  ONLINE       oimsec

ora.oimap.db

      1        ONLINE  OFFLINE

ora.oimpri.vip

      1        ONLINE  ONLINE       oimpri

ora.oimsec.vip

      1        ONLINE  ONLINE       oimsec

ora.scan1.vip

      1        ONLINE  ONLINE       oimpri

ora.scan2.vip

      1        ONLINE  ONLINE       oimsec

ora.scan3.vip

      1        ONLINE  ONLINE       oimsec

 Database cluster resource ora.asdb.cfcdb dependency tree

——————————————————–

 

   Basic information

————————+————————

  Name                  |  ora.asdb.cfcdb

  Type                  |  Database Instance

  Target state          |  ONLINE

  Resource state        |  ONLINE on oimpri

  Restart count         |  0

  Failure count         |  0

  Hosting members       |  oimpri, oimsec

————————+————————

 

  Common CRS parameters

————————+————————

  Auto start            | Yes

  Check interval        | 120 sec

  Failover delay        | 30 sec

  Failure interval      | 700 sec

  Failure threshold     | 5

  Restart attempts      | 4

  Script timeout        | 300 sec

  Start timeout         | 300 sec

  Stop timeout          | 300 sec

————————+————————

 

  Resource specific parameters

————————+————————

  Database component    | dbinstance

  ORACLE_HOME           | /dbfiles/app/oracle/product/11.2.0/db

  ORACLE_SID            | oimap

  pfile                 | Database default

————————+————————

   Resource dependency tree(s)

————————————————-

  ora.asdb.cfcdb

    |

    +->ora.asdblsnr.cfcdblsnr

    |    |

    |    +->ora.dbvip.cfcvip

    |    |

    |    +->ora.dbdisk.cfcdisk

    |

    +->ora.dbdisk.cfcdisk

 

 e cluster resource ora.obieeserver.cfcas dependency tree

——————————————————–

 

  Basic information

————————+————————

  Name                  |  ora.obieeserver.cfcas

  Type                  |  WebLogic Domain

  Target state          |  ONLINE

  Resource state        |  ONLINE on oimsec

  Restart count         |  0

  Failure count         |  0

  Hosting members       |  oimpri, oimsec

————————+————————

 

  Common CRS parameters

————————+————————

  Auto start            | Yes

  Check interval        | 50 sec

  Failover delay        | 20 sec

  Failure interval      | 350 sec

  Failure threshold     | 5

  Restart attempts      | 4

  Script timeout        | 600 sec

  Start timeout         | 600 sec

  Stop timeout          | 600 sec

————————+————————

 

  Resource specific parameters

————————+————————

  Component home        |

  (WebLogic domain)     | /fmwfiles/app/oracle/obieehome/user_projects/domains/bifoundation_domain

————————+————————

  Components

  (Managed servers)

————————+————————

  Name                  | Monitor

————————+————————

  AdminServer           | default

  bi_server1            | default

————————+————————

 

  Resource dependency tree(s)

————————————————-

  ora.obieeserver.cfcas

    |

    +->ora.obieevip.cfcvip

    |

    +->ora.obieedisk.cfcdisk

 

[root@oimsec bin]#

 

Advantages

1.Automatic recovery of Database and OBIEE resources from Node failures.

2.Lost Cost : License

3.Can be effectively used in environments like where cost is a constraint but scaling can be compromised. So this setup is best suited for small to  Mid-level customers.

4.Planned Maintenance: HW can be upgraded while the application is up and running, like Memory upgrades – both nodes.

5.High availability with 2 Tier setup (scaling) at reduced cost.

 

Disadvantages: 

1.May not possible to completely switch over or fail over the ASCRS resources  when  there some dependencies active.

So either manual intervention or some sort of automatic scripts to be in place in such scenario.

2.OBIEE sessions may need to refreshed or reinitiated after the Failover or switch over.

Oracle is aware of these complexities and still supports this architecture for low to  mid- level customers (Cost).

Please remember it is possible to build (oracle supported)  100% HA (both DB and OBIEE)  + Scaling (Vertical or Horizontal) in place  for OBIEE (or any type of Fusion Middleware application) component – at the expense of COST.

3. At present ASCRS is supported only in Unix platforms. 

 

Conclusion:

In Our Real Practical DR Scenario :

Switch Time

Database Node Crash – Database automatically migrated to OBIEE – OIMSEC Node

1 Min 20 Sec

OBIEE Node Crash – OBIEE automatically migrated to Database Node – OIMPRI Node

3 Min 10 Sec

Manual Failover the Database to OBIEE(OIMSEC)Node – Maintenance

1Min  40 Sec

Manual Failover the OBIEE  to Database (OIMPRI) Node – Maintenance

2 Min 50 Sec

 

 

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s