Embedded Chips and the Year 2000

Washington Technology Online
http://www.wtonline.com/
16 May, 1997

Federal agencies now struggling to find funds to fix their year 2000 software conversion problem are facing another hefty bill: the cost of fixing faulty software embedded in health-monitoring devices, navigation equipment, elevators, sprinklers, air conditioners and other electronically controlled necessities of office life.

Rep. Steve Horn, R-Calif., said he will soon step up pressure on 30 agencies to estimate the cost of fixing these additional year 2000 problems - and then to begin actually fixing them.

Although no agency has investigated this problem, the price tag could rise above $2 billion, said Russell George, Horn's chief assistant. George serves as staff director for Horn, who is chairman of the management, information and technology subcommittee of the House Government Oversight and Reform Committee.

The extra cost of fixing the embedded software - especially software included in the Department of Defense's high-tech weaponry - will put an extra burden on agencies' budgets. That's a major concern for industry executives who fear that the contracts they hold - or expect to win - may be slashed to fund year 2000 fixes.

Software-drive chips [that control and monitor dates] are in "5 percent of literally everything from garage alarms to telephones to elevators," said one congressional staff member.

Agencies "will come up with a number because they are ordered, but it will be worthless," said James Kerrigan, president of Colmar Corp., a consulting firm based in Reston, Va. "I don't think it is possible to estimate their exposure because nobody has maintained an adequate inventory of these embedded systems," he said.

"Embedded [software] is the big unknown in all of this [year 2000 problem]. You have more manufacturers and suppliers involved," said Bob Cohen, vice president for communications at the Arlington, Va.-based Information Technology Association of America, which lobbies on behalf of its 350 corporate members, including Microsoft Corp., Redmond, Wash.; IBM Corp., Armonk, N.Y.; and Electronic Data Systems, Plano, Texas.

So far, agencies have been trying to identify and fix faulty software in mainframes and networked computers. Fixing these nonembedded systems could cost anywhere from $10 billion to $15 billion, Horn said.

Horn's estimate stands in sharp contrast to the White House's February budget request for 1998, which suggested the year 2000 problem would cost the government as little as $2.3 billion to fix.

The year 2000 problem was created by software designers' traditional exclusion of the century from their two-digit calendar codes. That becomes a problem when a computer is required to handle dates in the 1900s and 2000s, ensuring that processing priorities and digital calculations get scrambled.

That problem is also built into the software used in many electronically controlled items, such as the navigation devices used by the Federal Aviation Administration for the nation's air traffic control system, the health care devices used by the Department of Health and Human Services, or the elevators, air conditioning and security devices used by every federal agency.

Thus, an elevator might shut itself down on January 1, 2000, mistakenly believing that the previous week's maintenance check never happened - or has already been performed 99 years into the future.

Some unknown percentage of the repair costs will likely be borne by the manufacturers as they maintain equipment, said industry officials. "Hopefully they'll fix the elevators, and hopefully they'll fix the phones," said the congressional staff member.

Also, the federal agencies' continuous purchase of office equipment, such as printers, with more modern products, should leave federal agencies with few pieces of office equipment prone to failure by 2000, somewhat alleviating the problem posed by embedded software, said Kerrigan. But industry's concern over losing funding to year 2000 fixes was fortified when Horn said the government should solve the problem by shifting funds from other projects. "That worries people, but it's the only way," he said May 2 at a Washington conference hosted by the ITAA.

Also, government executives are considering ways to pressure vendors to pay for fixes to software they supplied years earlier, said Nancy Peters, CACI International Inc.'s vice president for strategic marketing and year 2000, based in Arlington, Va. Agencies frequently have software maintenance contracts with companies, which will perhaps force companies to pay the unexpected cost of the year 2000 problem, she said.

"Some companies may have to give up a little. It is a contract-by-contract situation," said Harris Miller, the ITAA's president.

"We are going to end up seeing some sort of shared liability," predicted Kerrigan.

However, some agency officials are adopting a less aggressive attitude and are more willing to acknowledge that "we're all in this together," Peters said.

But few industry or government officials are optimistic that the agencies can fix the problem in time. "Until we have a catastrophe or two, you may not get the top people on board" the effort to quickly fix the problem, said Joel Willemssen, director of the Information Resources Management division of the General Accounting Office, the audit arm of the U.S. Congress.

Moreover, top federal managers "are terrified by [the year 2000 problem]. They're planning to retire before it hits, if they can," said Cynthia Warner, year 2000 program manager at the General Services Administration.

Date: Fri, 25 Apr 1997 22:10:18 -0700

From: gielan 

To: year2000-discuss@year2000.com

Subject: Re: Automobiles: Embedded systems



Folks,



Lets try and separate the wheat from the chaf when addressing

microprocessor controlled devices vis-a-vis the Y2K problem.



IMHO, no _year_ related issues will appear in a substantial portion of

the microcintrollers being discussed here. Place yourself in the

position of a hardware engineer, designing a processor based controller

(even if it IS an 80x86 type), and writing the software for it. If you

need timing issues, the LAST thing you would use is any form of calendar

for that purpose. You would use an integer counter, probably 32 bit,

counting seconds, or hours, or whatever, based on some crystal oscilator

providing some counteddown base frequency. Your Real Time Clock chip

would probably be just a counter connected to the oscilator, delivering

to you a time base.



As such, it would NOT matter what the year is at all, and we should put

this issue to bed. Cars do not care if it is Monday, or Tuesday,  April

or May, year 22 or 1304958. They care about 352,900 (or some such

number) seconds since last maintenance, and how long was your trip (if

you have a trip computer), and what is your gas mileage (which is

dependent on distance, time difference in seconds, and gas consumption

rate), and when and how much to advance your electronic timing

(depending on your elevation, the throttle position, the torque, load,

etc.). And if your car DOES have a date display, and it is wrong, WHO

CARES!!! it will still run.



Where DOES the calender matter, then?  It matters whenever a function is

dependent on the day of the week or month, such as in elevators (weekday

or weekend or holiday schedule), alarm systems (ditto), VCR's (and only

a fraction of those on the market), some digital watches (which MAY

think they are in the 20th century in 2000, and misdisplay day of week

for date). So far, nothing life threatening.



The problem arises when the wonderful (?) invention called the PC (not

all personal type computers fall into that category, but most of those

in existence in the world do) is embedded _together_ with its software,

and some support chips, that the problem manifests itself.



For the basic problem, you need to look at 3 components of the PC:



      1. The software which is "burned-in" to a Read Only

         Memory (ROM) chip, commonly referred to as BIOS;

      2. The format of information stored (by the BIOS) in

         the battery backed-up Random Access Memory(RAM),

         commonly referred to as the CMOS;

      3. The output of the Real Time Clock (RTC) chip, which

         can be probed to deliver the components of the

         calendar (day of week, month of year, year, julian

         date, etc.)



So, any embedded or even attached system which is based on the common

design of the PC, has already 3 weak points, even before ANY software is

installed - DOS, Windows, Access, Word, Excel, or your own (or

purchased) special purpose application program. If you have a control

unit of any kind which includes an EMBEDDED PC, beware. If it is

attached, likewise. But if the controller is NOT based on the PC and its

BIOS etc, but a custom design, it will probably be OK.



To clarify, the 80x06, Pentium, PowerPC, or Alpha chips themselves (to

name a few), or the typical controller style chips (8049, 8051, etc.),

are no more likely to care what the year is, than your Serial port chip

in your modem.



I hope this clears some of the misconceptions, and is, in no way shape

or form intended to relax the concern about REAL potential failures of

PC's  - Karl has a list an arm long of those, and his list is still

growing.



Dan Gielan

Chief Technical Officer

VAST Corporation

(703) 821-3390

The terminology varies: embedded chips, non-compute devices, black boxes. To many of us who have programmed since the age of punch cards these chips have been and remain a bit of a mystery. Somehow, in spite of their lack of an operating system, compiler, and other things with which we are familiar, they none the less somehow act computer-like.

From the early days of computing these chips have been in the domain of the engineers, and we of the programming world know these guys are different. No - they don't have 8 fingers on each hand, but other than that the two professions have had a long-standing (and long distance) distrust for each other.

As we began the search for potential problems that will occur after December 31, 1999, a number of non-compute devices were immediately identified ... fax machines that won't print 01/01/2000 on the top of incoming messages, PABX's that interface with PC's to print monthly reports with mm/dd/yy dates, and VCR's that the manual said had a date function but we stopped short of trying it after we finally succeeded in making it stop blinking 12:00.

Suddenly we looked around, and there are non-compute devices everywhere! They are in our air conditioners and furnaces, automobiles, elevators, burglar alarm panels, and traffic lights. Looking further we discovered them in process control systems ranging from nuclear power plants to labeling equipment in dairies and potato chip factories. Then we become officially involved, as management declares that "Year 2000 compliance is going to be supervised by our IT people".

Rumors sprang up and were spread quickly, spreading more questions than answers. Why? The vast majority of these non-compute devices are truly black boxes from an end user's viewpoint. We cannot go to a keyboard and test a year 2000 date roll-over. We cannot list the code on a screen or keyboard. I have tried and cannot even find more than 3 of the 11 embedded chips I am told run my pick-up truck! Adding fuel to the fire were the vendors, who generally gave us a blank stare when querried on Year 2000 compliance, and then stated with great assurance that it isn't a problem with their product.

I don't claim to have very many definitive answers. I really cannot say with "tested assurance" that your 1994 Honda Civic will or won't run after the roll-over. What I have done is assemble the results of many frustrating hours looking for answers with some words of wisdom gleened from the Year 2000 list-serve. I have bypassed discussion of the GPS (Geo-Positioning Satellite) system, which is covered in another paper on these pages.

Gerry Docherty      ged@rtel.demon.co.uk

Embedded systems and real-time problems



Thanks to all who have recently contacted me on the above topic.

Apologies for the slow response - it's been a bit  hectic recently!

The following is a general chat on the problems peculiar to embedded

systems. If you're working with such systems just now, you should

recognise most of them.  I've enclosed a copy of the presentation

slides I've been using recently - I can't claim that they are

entirely self-explanatory, but I hope they help.



There's been some renewed discussion recently about Year2000

problems as they relate to real-time and embedded systems in

production, manufacturing and engineering environments. We're

working on resolving the Year2000 problems in these areas already,

delivering the Real Time 2000 Programme to a variety of high-

profile clients. I hope the following is a helpful contribution.



The simple fact to grab and hold onto is that embedded systems

underpin the whole of the world's manufacturing and engineering

base. The world's energy supplies (oil, gas, coal, nuclear) depend

on embedded systems.  Planes fly, and ships sail, based on embedded

systems. Pharmaceutical industries use embedded systems to create

the world's drug supply. The food we eat, the drink we consume,

primarily comes from processes which depend on embedded systems.

Not to mention clean water. And, of course, defence of the realm is

heavily based on embedded systems. And car manufacture. And railway

networks. And broadcast media.  And communications. And so on.



So real-time and embedded systems are prime components of global

infrastructure. They are also the commercial building blocks of

engineering and manufacturing worldwide. So addressing the Year

2000 problems for these systems is at least as important as doing

it for banking and financial institutions. Probably more so. And

fixing the problems is more complicated.



It's true that all of the problems which exist in traditional big

IT applications also exist in real-time and embedded systems. So

we can have problems arising at processor level, or from operating

systems, packages/tools, and bespoke applications. The technical

solutions are also much the same - some replacements, some

modifications, some workarounds.



But the big difference is the culture which surrounds real-time or

embedded systems in production environments. Real-time systems can

be very complex, and they are used to control or monitor very high-

value processes. Typically, a large installation (e.g., a petrochemical

refinery, oil/gas platform, power station) will have scores of

real-time systems. They have been bought for different reasons by

different people over the years, usually mirroring the gradual

development of the installation. The production processes  are now

dependent on the successful continuous operation of the real-time

systems.



Because the production processes are so valuable, production managers

and engineering staff fear the failure of real-time systems. When

real-time systems fail, high-value processes shut down, and the costs

of unexpected shutdowns can be enormous. For oil platforms,

pharmaceutical manufacturers or power stations, the cost of an

unexpected shutdown can be hundreds of thousands of pounds. Even for

small manufacturing companies, the costs are crucial, because the

production process is their only true source of income.  The pressure

to keep the production process running is great. As a result,

production managers resist changes to embedded systems on the

if it ain't broke, don't fix it basis.



This means that when the next version of the operating system comes

along, it is not automatically installed. If improved functionality

could be achieved by upgrading bespoke software, it is not acted upon.

Hardware which is no longer supported by the manufacturer remains in use.

The result is a bunch of ageing systems, based on languages, packages

and processors for which the skills are gradually being lost.  Because

of this culture, fixing the Year2000 problems is more complicated than

for banking or a dministrative applications. The systems are more

difficult to audit, because some are so old that the information about

them has literally been lost. Systems dating from the late 70s and early

80s are pretty common.  Doing the triage is complicated, because there is

a risk that taking the system through a mock millennium change will cause

the process to fail, with huge cost penalties. Applying the fixes is

fraught, again because of the potential to cause a production failure.



So to fix the problems, you need people who understand embedded systems

technology, the production processes, and the commercial impact of

mistakes in a manufacturing environment. These people are very, very

thin on the ground. There are not many companies who specialise in

real-time and embedded systems.



From what we can see, few manufacturing companies have recognised the

scale of the problem yet. Systems are not yet failing, because

real-time systems tend to have a lookahead of less than a month. So

the failures will come late in 1999. Nonetheless, from our work over

the past six months in this area, we know that the likelihood of

failure of embedded systems is high.  The companies we are working

with are in the vanguard. The big organisations might be able to sort

themselves out by throwing money at the problem, though resources will

be very scarce. The small manufacturers are in trouble - most of them

don't know they have a potential problem, and when they find out,

they'll find it very difficult to compete with the big boys for decent

skilled staff.  Remember, of course, that around the office, embedded

systems are widespread - personnel tracking systems, PABX and Fax

machines, security access, heating and air conditioning, etc. Outside

the office environment, humble domestic appliances, alarms systems,

video recorders and the like also use embedded systems. For the most

part, the failure of these systems will have nuisance value, and I

don't worry too much about that. Between here and the Year 2000, we

only have time for the important problems.



Gerry Docherty, Real Time Engineering Ltd., Academy House, Academy Park,

Glasgow G51 1PR, United Kingdom. +44 141 427 4142.  ged@rtel.co.uk

"  Apart from misgivings about missing out on New Year parties as the last

seconds of 1996 ticked away, it was much like any night for the staff at the

Tiwai Point Aluminium smelter in Southland.



   Midnight and crisis struck in the same moment.  Each of the 660 process

control computers that run the smelters potlines hung, their digital chips

frozen.



   Five pot cells were ruined, leaving New Zealand Aluminium Smelters with a

repair bill estimated at more than $1 million (NZ), though the company will

not confirm the final cost.



   What happened?



   It became clear only when Comalco's Bell Bay smelter in Tasmania shut

down precisely two hours later: 1996 was a leap year.  Unable to cope with

an extra day, the computers at Tiwai Pt (New Zealand) and Tasmania (Australia)

stalled with expensive consequences"

Date: Fri, 16 May 97 15:32:41 EDT

To: buytexas@swbell.net     kwf@gmt-2000.com

From: "JANET FLYNT"   flyntj@ul.com



Hello,



I recently learned about some discussion regarding UL and UL's capability

to address the Year 2000 problem that involved both of you.   So here are

the facts about UL's efforts in software and how we are viewing the Y2K

problem.



 In January of 1994,  UL published the First Edition of UL 1998 Standard

for Safety-Related Software.  We have been conducting reviews using this

Standard and have a growing number of software engineering professionals at

UL to provide these third party reviews.   Additionally,  UL 1998 is being

reviewed under the ANSI review process and we plan to have a Bulletin out

with the Proposed Second Edition in July of this year. The Proposed Second

Edition will have a new name -- Standard for Software in Programmable

Components.



Please note that I used the word "review" and didn't use the word "test".

Due to the diversity of software functions and the application-specific

nature of testing programmable components, UL 1998  indicates neither

testing protocols nor tools.  Instead, UL 1998 contains requirements that

define test objectives and criteria for the general case.  This permits the

applier of the standard to choose from many testing protocols and tools as

long as the test objectives and criteria are met.



With respect to the Y2K situation,  the impact of the date change on

product safety will vary based on product functionality and the software

design and implementation.   Thus,  instead of making statements about what

products are or are not effected, we prefer to address the specific impact

on an individual product basis.    To this end, the UL 1998 Standard  for

Safety-Related Software has requirements that address risk identification

as well as design and testing requirements. These requirements can be

applied to the date problem as well as any  problem where an inadequate

number of bits are available to represent safety-related information.    We

are  incorporating  review questions in our Programmable Systems

Certification reviews that would identify the potential safety impact of

Y2K dependencies.



As the Primary Designated Engineer for UL 1998 and the Technical Lead for

our Programmable Systems Certification Services,   I'd be happy to discuss

the potential for application of  the UL 1998 Standard to the Y2K

challenges that you are faced with.



Cheers,



Janet S. Flynt

919-549-1765 (Phone)

919-547-6111 (Fax)

flyntj@ul.com

There is, however, a huge number of multi-purpose computers hidden in mechanical rooms or behind building control panels or wired into the backplane of process monitoring panels. These are in black boxes that do not look like PC's, 370's, or 1170's, but are subject to many of the soft spots you so correctly attribute to the PC's. Worse, a good deal of their functionality is precisely to report, act upon, or otherwise use the formatted date and day-of-the-week ... exactly those functions often left out of embedded chip programming.

These computers often run only one program, they run it very well and with a high degree of stability, and even reboot and resume running after power shut-downs. They smoothly regain control of the embedded chips they are charged with governing, and life resumes.

Question: Will a multi-purpose computer posing as a black box continue to run its program after the formatted date it is producing becomes nonsense? Will the embedded chip continue to perform its more mundane task without guidance from its date-aware big brother? Will the computer program crash, halt, give slightly wrong answers, or absurdly wrong answers? Generally, will life pretty much go on?

In some cases, the answer is certainly 'yes', as in the probable case that an elevator moving from floor 22 to 1 at ultimate midnight will remember where it was going. In some, the answer is without a doubt 'no', as in the case of a room entry control panel which is not to let anyone into a secured area on week-ends or holidays, but depends on the building management computer to tell it what day it is or, more commonly, "ok" or "not ok" to open.

The problem with embedded chips is, I believe, very similar to what is causing a great deal of real apprehension in the legacy computer world ... communication! Generally the problem is not a single program or even a set of programs and their associated databases. Rather, the problems only become non-trivial when more than one entity is involved, and communication is required to achieve the desired results. Can we hope that a building management controller will correctly communicate with its set of chips much better than Visa and Master Card have done getting banks to handle expiration dates of '00'?

There is a really extensive, interesting, and well written page which describes virtually every microchip commercially produced. In addition to being an invaluable resource, its presentation makes for good reading.
The Great Micro Processors of The Past and Present.

For comments, problems, and software submissions regarding this web site Contact Us..
Microsoft, MS, and Windows are registered trademarks of Microsoft Corporation. Microsoft Corporation in no way endorses or is affiliated with Window95.com. All other trademarks are the sole property of their respective owners.
Updated daily.