Day light saving : Prod outage : server shut down / FAILED

If anyone of  you faced the issue of server getting stopped automatically yesterday (Sunday, March 12, 3:00 am), this could be due to Day light saving. Error log as below:

soa_server1.log:####<Mar 12, 2017 3:00:02 AM CDT> <Critical> <Health> <xxxx> 
<soa_server1> <[ACTIVE] ExecuteThread: '35' for queue: 'weblogic.kernel.Default 
(self-tuning)'> <<WLS Kernel>> <> <xxxxssdd:15a91b0a340:-7ffd-0000000018e09e53> 
<zzzzzz> <BEA-310006> <Critical Subsystem JTAMigratableGroup has failed. 
Setting server state to FAILED.

The leasing table won’t be updated with proper time and any gap of more than 1 hour would mark the corresponding server dead. This caused SOA_SERVER1 to shutdown itself.

This symptom is mentioned in  Doc ID 1590774.1.

Solution: Patch 17033308 is available for this issue.

SOA Performance Tip : Reduce the MDS size !!!!

Generally, when there is a case where the heap usage is more and you are getting back-to-back full GC in Prod, this is really difficult to handle in production.
When getting into details, MDS is one of the components that is very heavy and already loaded in memory. As a best practice, all XSDs will be stored in MDS which is generally heavy in size, unless they are non-standard xsds. These xsds are having a lot of documentation around it which makes it more than the actual definition. Obviously these documentation help for the reason of better understanding

Tip : See if these documentation can be removed to reduce the size of xsd. This can be done easily with a very powerful tool/utility in Linux. It is the Sed  command. Sed is the ultimate stream editor command that can do wonders.

Lets take an example of such xsd and see how can sed command be used to remove the documentations/enumerations of a xsd file

sed -i '/xsd:documentation/d' sample.xsd
sed -i '/xsd:enumeration/d' sample.xsd

Here you go !!!!… check the size of xsd to see the size getting reduced after removing the documentations/enumerations
Save this modified file and import them back to MDS

Oracle B2B : Performance tips – 2

This tip is again related to XEngine !!!
As Oracle B2B is a multi protocol engine catering to messages exchange with trading partner. The widely used standards are X12, EDIFACT, HL7. HL7 can also be used by another product – SSHI (SOA Suite for Healthcare). All these (X12, EDIFACT, HL7) are using the XEngine to translate the message from Native-to-XML and viceversa.
Along with the document definition (ecs file), XEngine also uses the parser schema to translate the message from Native-to-XML and viceversa. These parser schema are present in one of the configuration file in the XEngine installation location (Oracle_Home\soa\soa\thirdparty\edifecs\XEngine\config\XERegistry.xml). As there are many parser schema of different standards are present in this file, the parser will go in a sequential manner to find the correct parser schema for every message !!!

Tip : is to keep the required parser schema in the top of the list, so that the engine finds in the first iteration and proceeds further, thus improves the processing time !!!. This means, if we are using the HL7 standard, place the HL7 specific parser schema on the top of the list as mentioned in below snippet

<Registry Name="XERegistry">
 <Product Name="Parser">
  <Category Name="ParserSchema">
   <Item Name="SchemaFile">${XERoot}/config/schema/HL7_parser_schema_FHS-NV-2.6.ecs</Item>
   <Item Name="SchemaFile">${XERoot}/config/schema/HL7_parser_schema_MSH-NV-2.6.ecs</Item>
   <Item Name="SchemaFile">${XERoot}/config/schema/HL7_parser_schema_BHS-NV-2.6.ecs</Item>
   ..........

Couple of important Notes:
• This needs the server to be restarted after the change
• This is not a documented information by Oracle. Hence, please get the acceptance by raising the SR, inorder for a continued support from Oracle

Oracle B2B : Performance tips - 1

Oracle B2B is a multi protocol engine catering to messages exchange with trading partner. The widely used standards are X12, EDIFACT, HL7. HL7 can also be used by another product - SSHI (SOA Suite for Healthcare). All these (X12, EDIFACT, HL7) are using the XEngine to translate the message from Native-to-XML and viceversa.

Consider a usecase where we have any of the documents involved (X12, EDIFACT, HL7) with moer than 30 definitions (ecs / XSD) used in the enterprise. it slowly impacts the performance of the message processing. This is due to the reading of ecs definitions to be read from MDS for each message exchange. However, XEngine has a property to cache the ecs definition where we can specify the number of definitions that can be cached. This is present in one of the configuration file in the XEngine installation location (Oracle_Home\soa\soa\thirdparty\edifecs\XEngine\config). Change this to the required number of unique count ecs that is used.

<Category Name="Settings">

        <Item Name="CacheSize">0</Item>

        .............   

</Category>

Couple of important Notes:

• This needs the server to be restarted/
• This is not a documented information by Oracle. Hence, please get the acceptance by raising the SR, inorder for a continued support from Oracle

Performance Improvement Tip for a BPEL project – Continued [IMPORTANT]

Having said on few of the tips on the previous post, there are some the tables still continue to persist the data which can still cause concerns. The Instance data occupies space in Oracle BPEL Process Manager schema tables. Data growth from auditing and dehydration can have a significant impact on database performance and throughput.

This occurs mainly when we have Async  BPEL process. The very reason for BPEL to persist few of the information like references, state of the instance, metadata, etc., are to cater to handle the faults and to recover the same.

Few of the tables are :
document_dlv_msg_ref : Stores references to dlv_message documents stored in the xml_document table.
xml_document : Stores all large objects in the system (for example, dlv_message documents). This table stores the data as binary large objects (BLOBs). Separating the document storage from the metadata enables the metadata to change frequently without being impacted by the size of the documents.
dlv_message : Stores incoming (invocation) and callback messages upon receipt. This table only stores the metadata for a message (for example, current state, process identifier, and receive date).
With this, there are high chances of the data growth in the tables and resulting in table space issue or the disk space issues.

One of the ways to handle this would be :

• incorporate the proper purge strategy ( Preferred )
• consider, revisiting the retention policy
• Disk space increase (Last option)