In September of 2001, the PICMG (PCI Industrial Computer Manufacturers Group) formed a subcommittee to define a next generation compute platform targeted at high throughput highly reliable applications. This subcommittee was composed of 105 companies representing users and manufacturers of industrial and telecommunications compute equipment. This group had the goal of creating, reviewing and releasing a new specification by the end of 2002. The specification development was a good example of blitzkrieg engineering in which multiple teams focus on individual parts of the problem with the hope that in the end all the groups will converge with a usable solution. A core group of individuals was responsible for monitoring the progress of the teams and to watch for potential incompatibilities and roadblocks. The end result of this 12 month effort is PICMG 3.0 Advanced Telecom Computing Architecture. The PICMG 3.0 specification is 460 pages long and consumed 5-person years of meetings and conference call time in the development process. This article is intended to provide some insight into the motivation behind the specification and an overview of the specification itself.

Goal of PICMG 3.0

The PICMG standards organization requires that any new specification development start with a SOW (Statement of Work). The SOW provides the guiding light that ensures the specification stays on target and does not meander too far for the original idea. The PICMG 3.0 SOW is as follows "The PICMG 3.0 specification will establish a cost effective, fabric based modular, interoperable and scalable architecture for converging communications and data networking applications. Mechanical, thermal management, power distribution and systems management will be defined in this "core" specification, but it will be fabric agnostic so as to support transitions in technology over time or by application. The focus for features will be telecommunications carrier grade [Reliability, Availability, and Serviceability (RAS)] oriented with a second order objective to accelerate the adoption in highly available data center applications. PICMG 3.0 and its subsidiary specifications will provide a solution for markets that are currently underserved by existing standard and proprietary approaches." The specification is clearly targeted at applications not served by the existing PICMG CompactPCI specification. In fact, most of the members of the PICMG 3.0 subcommittee have roots in the CompactPCI ecosystem and were looking for a new platform that would be used to solve next generation communications and data applications. The CompactPCI community has tried to adapt PICMG 2.x standards to the telecommunications market requirements with limited success. In the end, the CompactPCI specification just did not have the board space, power dissipation, bandwidth or system management that telecommunications applications required. A new specification was needed.

A Family of Specifications

The title for the PICMG 3.0 specification is Advanced Telecommunications Computing Architecture or ATCA. This can be somewhat confusing as ATCA is used to refer to a family of specifications of which PICMG 3.0 is one of. PICMG 3.0 is different from other PICMG specifications in that it is made up of a core specification (PICMG 3.0) and a series of subsidiary specifications. The core specification defines the mechanical, power, cooling, interconnect and system management aspects of the ATCA family of specifications. The subsidiary specifications overlay a transport onto the interconnect defined in the base specification. In essence, the base specification defines the point to point connections used for boards to communicate and the subsidiary specifications define the protocols used over those interconnects. At this time 4 subsidiary specifications have been approved; 3.1 Ethernet and Fiberchannel Transport, 3.2 InfiniBand Transport, 3.3 StarFabric Transport, and 3.4 PCI Express Transport. An additional subsidiary specification PICMG 3.5 Advanced Fabric Interconnect / Serial RapidIO Transport is under development. The flexibility provided by the subsidiary specifications comes at a cost namely ease of use. An integrator of ATCA products will need to pay attention to the subsidiary specifications used by boards that populate the platform. A board that uses PICMG 3.1 Ethernet and Fiberchannel Transport and a board that uses 3.2 InfiniBand Transport will not be able to communicate over the fabric with each other.

Form Factor Overview

PICMG 3.0 mechanical requirements are largely derived from space utilization studies performed with 600mm ETSI and 19" EIA standard cabinets. The subcommittee wanted the most efficient utilization of the cabinet space coupled with a board size that would support the physical and thermal requirements for future generation components. The initial work was based on the Euro Card standard and then enhanced to support low cost sheet metal construction which was referred to as Simplified Telecom Packaging. The support of rear transition board was also highly desirable for applications that required rear I/O. In determining usable board space front and rear cable bend radius were also taken into consideration. After debating the merits of different board form factors the group agreed to an 8U x 280mm deep 1.2" pitch front board and an optional rear board 8U x 70mm deep. Based on a 1.2" board pitch, a shelf in a 19" EIA cabinet could support 14 slots and a 600mm ETSI cabinet could support 16 slots. A side view of ATCA front and rear boards in a 600mm ETSI shelf are shown below.

Side View of Front and Rear ATCA Boards in ETSI Shelf

Power Overview

PICMG 3.0 boards can dissipate up to 200 Watts. Although 200 Watts might seem a little on the low side based on silicon today, a shelf of 16 boards at 200 watts per board would need to dissipate 3200 Watts. A cabinet made up of three shelves would then have to dissipate almost 10K Watts of power. Boards that require more that 200 Watts would need to be double wide boards. These boards could take advantage of the cooling that two slots provide. With a power dissipation target of 200 Watts it was no longer feasible to bring power onto a board at low DC voltages - consider 3.3Volts at 60 Amps. The standard for telecommunications equipment is 48Volts which works out to be a comfortable 4 Amps at 200 Watts. In most telecommunications 48V DC power sources are currently available eliminating the need for a shelf level power supply. Dual redundant power feeds are used to eliminate the single point of failure that one power feed would have. The power connector contains an alignment pin to ensure proper alignment of a PCIMG 3.0 board and also contains pins for system management.

PICMG 3.0 Power Connector

Transport Overview

PICMG 3.0 contains multiple transports. These transports provide system management, control plane, and data plane connectivity. System management is based on a two wire serial interface (I2C) that is bussed between all slots. The system management messages are based on IPMI and extended for use in PICMG 3.0. The system management bus is redundant allowing management communication if one of the I2C busses fails. The next data transport is referred to as the Base Interface. The Base Interface is a dual star topology with redundant switch cards and Ethernet BASE-T signaling. The Base Interface is intended to provide an IP transport for PICMG 3.0 boards and is very similar to PICMG 2.16 in terms of architecture. The high speed data transport in PICMG 3.0 boards is the Fabric Interface. The Fabric Interface is based on 3.125Gbps SERDES signaling and is capable of supporting 10Gb data rates in a star or full mesh topology. As mentioned previously, the PICMG 3.0 specification defines the signaling and interconnects for the Fabric Interface; subsidiary specifications define the protocols that are used on these transports. The interconnects of a PICMG 3.0 backplane that supports a full mesh Fabric Interface, dual start Base Interface, and redundant system management busses is pictured below.

Summary

The PICMG 3.0 specification is the largest specification every created by PICMG. At 460 pages in length, it is a formidable document. This article was intended to provide a basic overview of the architecture and did not touch on the Ring and Test busses, Update Channel, Synchronization Clocks or System Management capabilities that are a part of PICMG 3.0. Although the PICMG 3.0 specification is released, the subcommittee is still holding weekly conference calls to review and resolve ambiguities that arise as vendors that were not involved in the development of the specification take a fresh look at the document. One other item to note is the amount of cooperation between competing companies in the creation of this specification. We had the best architects and engineering talent from many competing companies sit down and discuss the future of telecommunications computing equipment. We were able to work together in the creation of a specification that in the end is a combination of the thoughts and ideas of a diverse set of individuals. It was truly a model for cooperation for our industry.