NML Cryogenic control system with OPC

( Last updated: Wed Nov 17 14:45:43 2010 By J. You )

Fault Recovery Procedure

Stop when the issue is resolved
Check if nml-cryo5 is responsive
Check if APACS OPC Device Server process is running on nml-cryo5
Reboot nml-cryo5 if necessary

If nml-cryo5 is not functioning, ACNET OAC can be configured to connect to nml-cryo2.
Modify database ADBS with the following commands (B. Hendricks x2448 or D. Finstrom x4688 can help if I am not available)

          update youj.opc_acl set active_flag = 0 where opc_server_name = 'nml-cryo5.fnal.gov'
          update youj.opc_acl set active_flag = 1 where opc_server_name = 'nml-cryo2.fnal.gov'

Restart NMLOPC OAC, which is running on DCE45. If N:NMLCTL is working, set it to 9999. Or contact MCR at x3721.
Contact J. You at x6682 or (630)7016788.

Here are four diagnostic ACNET devices that can be used to check if NMLOPC is healthy. They are available on H55/1.

N:NMLOPC, returns NMLOPC status code. 0 means OK.
N:NMLCTL, control device for NMLOPC
N:NMLUPT, returns the uptime of OAC in minute. This is updated every 3 minutes. Correct reading indicates both reading and writing to APACS are successful.
N:NCRR1, returns a counter from APACS. The counter is incremented once a second, roll-over at 3600 and served as a guage for timing performance.

Sample Dabbel templates for ACNET device setup

Here are some sample dabbel templates for adding/updating ACNET devices.
Please note: OAC is required to re-start for the change to take affect.

Device that returns float value

!
ADD N:NCRR1  ( "Read Real block1 status ", NMLOPC,         , 01FFFFFE, 0 )
SSDNHX READNG (0032/0000/0000/0000)
PRO READNG ( 4, 4, 60)
PDB READNG ( "STS ", "STS ", 16, 4, 4, 0, 0, 0, 0, 1)
FMAP READNG ( "APACS",
  "NML1.|READ_REALA|", "Float", 0, 1)
!

Device that has both reading and setting properties

!
ADD N:NHTSP  ( "NR Dewar HeatEr LOOP SET", NMLOPC,         , 01FFFFFE, 0 )
SSDNHX READNG (0032/0000/0000/0000)
PRO READNG ( 4, 4, 60)
PDB READNG ( "%FUL", "%FUL", 16, 4, 4, 0, 0, 0, 0, 1)
FMAP READNG ( "APACS",
  "NML1.HEX_N.VALVES.LLDEWH_SP", "Float", 0, 1)
SSDNHX SETTNG (0032/0000/0000/0000)
PRO SETTNG ( 4, 4, 60, 00, 00, A8, 41)
PDB SETTNG ( "%FUL", "%FUL", 16, 0, 4, 0, 1, 0)
FMAP SETTNG ( "APACS",
  "NML1.HEX_N.VALVES.LLDEWH_SP", "Float", 0, 1)
!

Device that is alarmable

!
ADD N:C2LLHD ( "SCRH LIQUID LEVEL PROBE ", NMLOPC,         , 01FFFFFE,
            "TvFrig" )
SSDNHX ANALBL (0032/0000/0000/0000)
PRO ANALBL ( 2, 20, 60, 00000000, 42DC0000, 2, 4, 1, 1,
             0, 0, 0, 0, 10, 0, 0)
PRO ANALTX ( 0, 0, 0, 0, " ")
SSDNHX READNG (0032/0000/0000/0000)
PRO READNG ( 4, 4, 60)
PDB READNG ( "%FUL", "%FUL", 16, 4, 4, 0, 0, 0, 0, 1, 0, 0, 0, 0)
FMAP READNG ( "APACS",
  "NML1.|C2_LT_LLHD|", "Float", 0, 1)
!

Device that returns integer

!
ADD N:NMLUPT ( "NMLOPC Uptime           ", NMLOPC,         , 01FFFFFE, 0 )
SSDNHX READNG (0032/0000/0000/0000)
PRO READNG ( 4, 4, 60)
PDB READNG ( "Min ", "Min ", 10, 4, 4, 0, 1, 0, 0, 1)
FMAP READNG ( "APACS",
  "NML1.|UPTIME|", "Integer", 0, 1)
!

Technical Overview

The NML cryogenic control system uses OPC client-server architecture. APACS OPC server runs on nml-cryo5. And the OPC client is implemented as OAC (NMLOPC) running inside DAE of the accelerator control system.

The client makes use of three open-source java libraries.

The NMLOPC has built in re-try logic to recover from issues like network disruption or APACS server not available. It will try to reconnect to APACS server for a maximum of 3 times. If it still fails to establish the communication, it will trigger an OAC restart process. Notification email is sent for each re-try.

APACS OPC Server configuration

OPC/DCOM configuration for APACS is described here and some DCOM configuration specifically for XP SP2 can be found here.

Other configurations might be needed:

Email MACK to have the "OPC" group policy apply to the new OPC server
The policy does the following:
Use RUN\gpedit.msc to add group "FERMI\Domain Users"
Local Computer Policy|Computer Configuration|Windows Settings|Security Settings|Local Policies|User Right Assignment|Access this computer from network
(This will cause error "Connection timed out" otherwise)
Use dcomcnfg to add "FERMI\Domain Users" to "Edit Default" and "Edit Limit". And DCOM=OPC.SIMATIC.NET.
(This will cause error "Access denied" otherwise)
Also check "Identity/Launching user"
DCOM uses ports TCP/135, TCP/139, TCP/445.
Configure "Windows Firewall". Add "APACS OPC Server, OPC.SIMATIC.NET" to the "exception list".
(This will cause error "RPC is not running" otherwise)
Make sure "Remote Registry" service running. See here
(This will cause error "Message not found for errorCode: 0xC0000034")

No specical software or account is needed for OAC.

NMLOPC OAC configuration

NMLOPC runs on DCE45. Parameters needed for OPC OAC is stored in database table opc_acl. Here are the table schema and sample data.

use appdb
go

create table opc_acl (
    oac_name        varchar (100) not null,
    opc_server_name varchar (100) not null,
    username        varchar (50)  not null,
    password        varchar (200) not null,
    prog_id         varchar (100) not null,
    phrase          varchar (50)  not null,
    alert_emails    varchar (200)     null,
    active_flag     bit           not null
)
go

delete from opc_acl where oac_name = 'nmlopc'

insert into opc_acl (oac_name, opc_server_name,
username, password, prog_id, phrase, active_flag, heartbeat_tag, update_freq_in_hz,
acnet_control_device, alert_emails)
values ('nmlopc', 'nml-cryo5.fnal.gov', 'serviceAccount', 'EncryptedPassword',
'apacs', 'phrase description', 1, 'NML1.|UPTIME|', '15', 'N:NMLCTL', 'youj@fnal.gov')
go

Other debugging information


/usr/sbin/lsof -u console | grep java | egrep 'pipe|eventpoll' | wc  # about 58 at startup.

Security, Privacy, Legal