Frig Front End Emergency Instructions

This page tells you what to do when you need a replacement PowerPC processor board for a Frig front end. Presumably, you are reading this because lightning has struck one of the Frig front end CPU cards or it has otherwise burnt-out or failed, and the primary Frig controls support engineer is not available.

Before proceeding with the replacement of the CPU card, here are some things you absolutely should have tried first (unless you see obvious signs of smoke, etc.):

• Reboot the processor. Try it again!
• Telnet into the card. See if it is working at all.
• Check for obvious things like cords unplugged or VME boards being seated properly.
• Take a terminal (e.g. vt220) out to the location, plug it into the PowerPC's serial console port and see if anything is working.
• Check the network. If things look like they are working on the console terminal, maybe the network or a network hub/router has failed. The network hubs for most of the zero buildings are relatively near the Frig VME crate. See if it appears to be working.
• Check the LEDs on the fronts of the carrier cards for the IPUCD and arcnet cards. The LED for each slot containing one of the IP cards should be pretty steadily lit up. If the LED for an arcnet card slot is blinking at 0.1 to 1 Hz, then it is having problems with the arcnet communications for that node. When the PowerPC has trouble communicating with only one house of the sector, it has proven to be a problem with the house's I/O or Thermometry crates nearly every time. Here are some things to check.
  • Reboot the 186 cards for the I/O and thermometry crates.
  • As a desparate measure, reboot the PowerPC after rebooting the 186s. This usually doesn't help.
  • Check the Power supplies for 186 cards for the I/O and thermometry crates.
  • Check/swap the 186 cards for the I/O and thermometry crates.
  • Only after having done the above (in our experience), consider replacing the arcnet hardware in the Frig VME crate. Start with the transceiver module which is mounted on the back of the VME crate.
• Check the VME crate's power supply.

• 1.

  Locate a spare PowerPC 2400 processor. You choices are
  • A. One already running Frig software, but configured for a different sector.
  • B. Other PowerPC 2400 board already configured for VxWorks.
  • C. Brand new PowerPC 2400 board. You need to install VxWorks and then this becomes the same as option B.
  You always want option A if it is at all possible. The most likely place to find such a processor is the Frig Test front end in the Linac Annex (first floor, straight in the door from the Linac hallway, VME crate is on the right).

  If it is a real catastrophe and you need a second spare, the PowerPC for FrigBR may be available. The frigBR system is generally only a test system, so you might be able to use it as a second spare. Check with the Cryo coordinator first. FrigBR is in the B0 service building. Make sure you take BR and not B0. They are labelled and also B0 has many more arcnet cards connected to it.

  If you need more spares than that, you'll have to go on to options B. and C., which are slightly more work (i.e. you have to get vxworks installed into the board for option C.).

• 2.

  Assuming you are using option 1.A. and the test frig board, you now need to change the configuration to that of the target sector (the one you are replacing). The configuration information for any Frig processor is stored in its FLASH non-volatile memory. This allows for a speedier, network-independent boot. Recall also that the Frig application code is also bundled with the VxWorks kernel and stored in ROM.

  The configuration information is also in another place: It's in the startup command scripts which are in the nova directory:

  /fecode-bd/vxworks_boot/fe/frig/ The command file names have the form X.login where X is the sector name: a, b, c, d, e, f, sy, pbar, and n2400 is the test system's name.

  You'll need to run the appropriate sector's command file to get that sector's configuration. Do the following to change the configuration:

  • Leave the processor (the one to be reconfigured) running the frig code.
  • Login to the processor (the one to be reconfigured) (telnet or console).
  • Find out where it's current working directory is:
    -> pwd
    The most likely answer will be:
    fecode-bd:vxworks_boot/fe/frig/defaults
    
  • Since we're going to run the command script, we want to start at the same directory where it the PowerPC starts after a boot. Go to the root:
    ->cd "../../../.."
    
  • Now run the command script for the sector desired. Note that the last thing in the command script is a command like:
    sp FrigStart
    

    That is the command which starts the Frig application code running. It is best to temporarily comment out that line for this reconfiguration. Remember to restore it when you are done.

    Run the command script like this:
    -> <vxworks_boot/fe/frig/a.login (or b.login, c.login, ...)
    

    There are several key components of this configuration, including setting the sector name, the number of houses, all the various addresses of the Arcnet and IPUCD cards, and reading the defaults files. Reading the Frig defaults files will take a minute or two to complete.

  • At the end of that command script, the configuration is in RAM, but not in FLASH. You must run the following command from the vxworks shell to write the configuration to FLASH:
    -> write_all_defs2flash
    
  • The FLASH reconfiguration is now complete. I recommend leaving the IP address and name unchanged one time and rebooting once. Note that the test node will try to access arcnets and devices which aren't connected since it is now configured as the other sector.
  • use the VxWorks bootChange command to change the "inet on ethernet", "gateway inet", and "target name". Also the startup script must be cleared for the operational sectors. (It is often set to some value for the test system for testing one thing or another.)

    The document frig_boot_params.txt contains the recent boot parameter settings for each frig node.

  • Replace the dead processor with the one you have just configured.

• 3.

  If you are forced to use option 1.B. (a PowerPC 2400 not already running Frig), then life isn't really too complicated. You need to perform the same basic steps:
  • Change the ethernet address, gateway, and target name.
  • Use the correct sector's startup script.
  • You can actually just set the startup script in the boot parameters and do nothing more and the PowerPC will function fine as a frig node. However, recall that the loading of the defaults files from the startup script (over the network) takes a long time (minutes). This makes the boot proces uncomfortably slow, but better than nothing.

    A better solution is to run the startup script, then execute:
    

    -> write_all_defs2flash
    Now change the startup script so that instead of calling global_load
    it now calls load_all_defs_flash
    This makes it read all the defaults out of flash instead of over the network.
  • Once you have configured the system, run the startup script one time, and run write_all_defs2flash, it is really a simple matter to load the "Frig application plus vxworks" code into ROM. All you need to do is run -> flashFile "../2400/vxWorks_rom.bin",870000
    (assuming you are in the usual Frig working directory of fe/frig/defaults).

    After running write_all_defs2flash and flashFile, just clear out the "startup script" boot parameter field, and reboot and everything should work. See the New Frig Build and Install Instructions for more details.