Moira Shire Council

1. The following error occurred during LIVE to TEST data restoration.

-> When I was doing an “awptest” the following error occurred.

 -> Using the Activity Monitor in SQL, someone has a query connected to that database from COBWIN10-5, which they haven’t exited, and it’s preventing me from getting exclusive access to restore the database.  

-> First, to check the activity, open the Activity Monitor as shown below. Please verify if the activity is being run by a user or if it is an automated internal system process.

If the error is only in awptest, focus on checking and managing the activity for awptest only. Do not kill processes related to authtest or bitest. Based on the database where the error occurred, review the activity under the corresponding process.

• If the activity is being run by a user, ask them to close it.

• If necessary, you can terminate the activity by right-clicking and selecting the Kill option.

If someone is running the process, kindly ask them to close the activity. If it's a system process, right-click on the process and select 'Kill it'. Once no other processes are holding the database, the restore should proceed successfully.

 

-> After killing the query, the awptest database was successfully restored.

 

-----------------------------------------------------------------------------------------------------------------

-> During the patch upgrade, there may be an automatic restart process. If it indicates that the process is already running, and you’ve already restarted the app pools and tasks on the API server, please check if the Authority TEST service is running by attempting to launch it.

If it's not running automatically, you’ll need to manually start it using the following command: wasservice –start test

 

------------------------------------------------------------------------------------------------------------------------------------ 

 

 -> So I have deleted the old file (21/04/2024) authlive and awplive from the backend, then processed the backup again.

--------------------------------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

 5. BIS backup - restores manually in LIVE: 

-> During the End of Year (EOY) CRM BIS Rollover 2025, the BIS-API component was skipped during the P42 patch upgrade. As a result, we encountered an issue during the rollover process.

Typically, we perform a database backup prior to the rollover. In case of any issues, we can manually restore the backup in the LIVE environment using the procedure outlined below.

-> The procedure is the same as the standard LIVE to TEST copy. However, in this case, instead of copying from LIVE to TEST, we need to manually restore the backup of LIVE to the LIVE environment. 

Please note: Below procedure for BIS backup (bilive) — not to authlive or awplive.

Take the manual "bilive" backup in LIVE before EOY and restore that data into current LIVE "bilive"

->The below error occurred because some Internal process has been running in the Backend, I can't kill or close the process:

Note: Since this is a LIVE environment, we do not support killing any processes during business hours. Therefore, we must wait until after business hours to perform such actions in the LIVE system.

I waited until after hours, then killed the necessary "bilive" process and completed the restore.
Before performing any restore, it's important to confirm with the Finance and Payroll teams that no payroll processing is in progress, particularly for the authlive and awplive databases.

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

6. Authority LIVE-Unavailable(server or 404 page error): Complaints from users-> Authority was very slow this morning, and some pages were getting "Internal server errors",

->404 Page error may occur shortly after the new patch upgrade. This is typically resolved by restarting the servers. If the issue persists, please follow the steps outlined below.

• So I restarted all three servers: SQL, API, and APP server. 
• I did WebSphere server as well,
• Finally, wasservice -stop and start from the command prompt as well. like below:

->After, still I got the following error:

CIVICA Confirmed:

• Go to "IIS Manager" from the taskbar->  application pools on the API server were stopped.
• Solution: Sometimes the application pools won't automatically start after a reboot. To check, log on to the COBAUTHAPI server and open IIS  from the task bar. Expand the sites tab.

 

-> The screenshot shows the auth.app.live pool in a stopped state. From the menu on the right-hand side, you can select "Restart".

For your reference: To check this to access the server log from the Server (or) access from the directory \\cobauthapp\d$\IBM\WebSphere\AppServer\profiles\live\logs\server1

By reviewing the server logs, we can identify the cause of the 404 error. Based on the error message, we can then determine the appropriate troubleshooting method.

----------------------------------------------------------------------------------------------------------------------------------------------------------

7. Note: After Patch P40, we are encountering a new JSON error during the upgrade to the latest patch. Below are the details related to the JSON upgrade:

-> Sometimes JSON updates successfully, but sometimes it comes with an error, so the pre-check is completed with an error.

This time need to run the bash command: "gen_master_Json -u" It will fix the JSON error

After again, continue running with "upgrade_authority"

-----------------------------------------------------------------------------------------------------------------------------------------------------

8. Prior to the P40 patch, CIVICA requested an update to .NET Core version 8.0 on the backend. During the update, some component files were missed, which led to issues and caused the LIVE environment to become unavailable after we upgraded the P40: 

#INC-29552

 -> error message occurred in the LIVE environment after patch upgrade:  "Terminated the Batch y (Or) N" (it happened due to .Net 8.0 version not being updated properly, some component files were missing)

-> CIVICA identified that the system outage in the LIVE environment during the P40 patch update was caused by missing .NET component files. Once identified, CIVICA resolved the issue by restoring the missing components on the backend.

-----------------------------------------------------------------------------------------------------------------------

9. Patch P42 Upgrade Success and .Net error faced

When I attempted to apply the patch in the LIVE environment for the first time, the upgrade completed with errors.

-

>When I attempted the patch upgrade a second time, it initially appeared successful. However, some errors later occurred in the LIVE environment, affecting certain modules.

After investigating with CIVICA, it was confirmed that during the second patch upgrade attempt, the BIS-API module the system was skipped for upgrading. CIVICA then manually upgraded the BIS-API module, which resolved the issue by automatically.

 

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

10. During Patch P43 in the Test environment:(#INC-30768)

-> During the patch upgrade, the following error occurred in the server: "Network connection closed unexpectedly" 

-> At the end, the Patch was completed with error:

->Reason: CIVICA confirmed that the bis_api component failed to be installed. CIVICA manually installed the component which will resolve this issue. After I attempted the "authority_upgrade"patch again and successfully upgraded.

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

11. Errors:

After patch upgrade, TEST (TEST, Train or Demo) Unavailable:

Sometime below error can occur:

-> During this situation, please follow the below :

First step: restart the APP, API and SQL server and check again if it's not working go to second step.

Second step: Open services in taskbar from COBAUTHAPP server -> Restart the IBM Websphere - check again the environmmet, If its not working move to 3 step

 Third Step: Pleae go to API server from the task bar open the "IIS manager" and to check the Application pool  -> Restart

Fourth Step: from API server, open "Services(Local)"  from task bar and please make sure the below service is active:

 After you did all the above procedures still has not been fixed the issue, still the environment is still unavailable.

-------------------------------------------------

Fifth step:  Especially during LIVE to TEST copy (TEST, TRAIN and DEMO) - CIVICA Incident - C00066939 

-> After I completed the LIVE to TEST & DEMO copy, a "404 page error" occurred. I tried all the above steps, still the error is still happening:

Why it's happened: The metadata for each non Live environment was pointing to Live. This was corrected by running setalparm -all and stamp_dbs in each environment to repoint the metadata. WebSphere was then restarted in each environment to apply the change. CIVICA suggested: It may be worth adding these two commands to your live to test procedure.

In this situation, run below additional commands in the Server

setalparm -all

 

stamp_dbs

If anything is not fixed the issue, please log into CIVICA with the server log error and system out.

Thanks,

Saranya