Phần cứng - Chapter 6: Storage systems

Computer ArchitectureChapter 6Storage SystemsProf. Jerry BreecherCSCI 240Fall 2003Chap. 6 - Storage*Chapter Overview6.1 Introduction6.2 Types of Storage Devices6.3 Busses - Connecting IO Devices to CPU/Memory. Interrupts etc. How is data transferred.6.5 Reliability, Availability and RAIDChap. 6 - Storage*IntroductionThe Big Picture: Where are We Now? We will look at how devices (especially disks) are put together. We’ll look at how to connect IO devices to the CPU.And then we’ll look at RAID, the brainchild of Patterson and his buddies.6.1 Introduction6.2 Types of Storage Devices6.3 Busses - Connecting IO Devices to CPU/Memory. Interrupts etc. How is data transferred.6.5 Reliability, Availability and RAIDChap. 6 - Storage*The Processor PictureChap. 6 - Storage*The Processor PictureProcessor/MemoryBusPCI BusI/O BussesChap. 6 - Storage*MemoryI/OThe Processor PictureRegistersCacheProcessorRegistersCacheProcessorRegistersCacheProcessorRegistersCacheProcessorChap. 6 - Storage*Types of Storage DevicesIn this section we will:Take a quick look at how disks work. This is only one example of IO, but we will save networks, tapes, etc. for another course.6.1 Introduction6.2 Types of Storage Devices6.3 Busses - Connecting IO Devices to CPU/Memory. Interrupts etc. How is data transferred.6.5 Reliability, Availability and RAIDChap. 6 - Storage*Disk Device TerminologyTypes of Storage DevicesPurpose: Long-term, nonvolatile storage Large, inexpensive, slow level in the storage hierarchyBus Interface:IDESCSI – Small Computer System InterfaceFibre Channel Transfer rate About 120 Mbyte/second through the interface bus. About 5 Mbyte/second off of heads. Data is moved in Blocks Capacity Approaching 100 Gigabytes Quadruples every 3 years  (aerodynamics)Can be grouped together to get terabytes of data.Chap. 6 - Storage*Disk Device TerminologyExample: Seagate Cheetah ST3146807FC147 Gigabytes10,000 RPM4.7 ms avg seek time.Fibre Channel$499.00Types of Storage Devices disks, 8 heads290,000,000 Total Sectors50,000 cylindersAverage of 6,000 sectors/cylinder or 800 sectors / track (but different amounts on each track.)MTBF = 1,200,000 hoursServerChap. 6 - Storage*Disk Device TerminologyExample: Barracuda Cheetah ST320822A 200 Gigabytes7,200 RPM8.5 ms avg seek time.ATA$299.00Types of Storage Devices disks, 4 heads390,000,000 Total Sectors24,000 cylindersAverage of 16,000 sectors/cylinder or 400 sectors / track (but different amounts on each track.)MTBF = ???????????? hoursDesktopThese are 4X more capacity than in 2001!!!Chap. 6 - Storage*Performance of Magnetic DisksSectorTrackCylinderHeadPlatter15,000 RPM = 240 RPS => 4 ms per rev Average rotational latency = 2 ms500 sectors per track => 0.10 ms per sector512 bytes per sector => 5,000,000 MB / sResponse time = Queue + Controller + Seek + Rot + XferService timeTypes of Storage DevicesRead CacheWrite CacheElectronics (controller)DataControlChap. 6 - Storage*BussesIn this section we will:Look at various bus mechanisms.In very simple terms, a bus is the connection between various chips/components in the computer.The bus is responsible for sending data/control between these various components.6.1 Introduction6.2 Types of Storage Devices6.3 Busses - Connecting IO Devices to CPU/Memory6.4 I/O Performance Measures6.5 Reliability, Availability and RAIDChap. 6 - Storage*Interconnect TrendsNetwork>1000 m10 - 1000 Mb/shigh ( 1ms)lowExtensive CRCChannel10 - 100 m40 - 1000 Mb/smediummediumByte ParityBackplane0.1 m320 - 2000+ Mb/slow (Nanosecs.)highByte ParityDistanceBandwidthLatencyReliabilityInterconnect = glue that interfaces computer system componentsHigh speed hardware interfaces + logical protocolsNetworks, channels, backplanesmemory-mappedwide pathwayscentralized arbitrationmessage-basednarrow pathwaysdistributed arbitrationBussesConnectsMachinesChipsDevicesChap. 6 - Storage*A Computer System with One Bus: Backplane BusA single bus (the backplane bus) is used for:Processor to memory communicationCommunication between I/O devices and memoryAdvantages: Simple and low costDisadvantages: slow and the bus can become a major bottleneckExample: IBM PC - ATProcessorMemoryI/O DevicesBackplane BusBussesChap. 6 - Storage*A Two-Bus SystemI/O buses tap into the processor-memory bus via bus adaptors:Processor-memory bus: mainly for processor-memory trafficI/O buses: provide expansion slots for I/O devicesApple Macintosh-IINuBus: Processor, memory, and a few selected I/O devicesSCCI Bus: the rest of the I/O devicesProcessorMemoryI/OBusProcessor Memory BusBusAdaptorBusAdaptorBusAdaptorI/OBusI/OBusBussesChap. 6 - Storage*A Three-Bus SystemA small number of backplane buses tap into the processor-memory busProcessor-memory bus is only used for processor-memory trafficI/O buses are connected to the backplane busAdvantage: loading on the processor bus is greatly reducedProcessorMemoryProcessor Memory BusBusAdaptorBusAdaptorBusAdaptorI/O BusBackplane BusI/O BusBussesChap. 6 - Storage*North/South Bridge architectures: separate bussesSeparate sets of pins for different functionsMemory bus CachesGraphics bus (for fast frame buffer)I/O busses are connected to the backplane busAdvantage: Busses can run at different speedsMuch less overall loading!MemoryProcessor Memory BusBusAdaptorBusAdaptorI/O BusBackplane BusI/O Bus“backsidecache”BussesProcessor Director Chap. 6 - Storage*Bunch of WiresPhysical / Mechanical Characteristics – the connectorsElectrical SpecificationTiming and Signaling SpecificationTransaction ProtocolWhat defines a bus?BussesChap. 6 - Storage*Synchronous Bus:Includes a clock in the control linesA fixed protocol for communication that is relative to the clockAdvantage: involves very little logic and can run very fastDisadvantages:Every device on the bus must run at the same clock rateTo avoid clock skew, busses cannot be long if they are fastAsynchronous Bus:It is not clockedIt can accommodate a wide range of devicesIt can be lengthened without worrying about clock skewIt requires a handshaking protocolSynchronous and Asynchronous BusBussesChap. 6 - Storage*° ° °MasterSlaveControl LinesAddress LinesData LinesBus Master: has ability to control the bus, initiates transactionBus Slave: module activated by the transactionBus Communication Protocol: specification of sequence of events and timing requirements in transferring information.Asynchronous Bus Transfers: control lines (req, ack) serve to orchestrate sequencing.Synchronous Bus Transfers: sequence relative to common clock.Busses So FarBussesChap. 6 - Storage*One of the most important issues in bus design:How is the bus reserved by a device that wishes to use it?Chaos is avoided by a master-slave arrangement:Only the bus master can control access to the bus:It initiates and controls all bus requestsA slave responds to read and write requestsThe simplest system:Processor is the only bus masterAll bus requests must be controlled by the processorMajor drawback: the processor is involved in every transactionBusMasterBusSlaveControl: Master initiates requestsData can go either wayArbitration: Obtaining Access to the BusBussesChap. 6 - Storage*The Daisy Chain Bus Arbitrations SchemeAdvantage: simpleDisadvantages:Cannot assure fairness: A low-priority device may be locked out indefinitelyThe use of the daisy chain grant signal also limits the bus speedBusArbiterDevice 1HighestPriorityDevice NLowestPriorityDevice 2GrantGrantGrantReleaseRequestwired-ORBussesOrder is:RequestGrantRelease.Chap. 6 - Storage*Even memory busses are more complex than thismemory (slave) may take time to respondit may need to control data rateBus RequestBus GrantCmd+AddrR/WAddressData1Data2DataSimple Synchronous ProtocolBussesClockChap. 6 - Storage*AddressDataReadRequestAcknowledgeMaster Asserts AddressMaster Asserts DataNext AddressWrite Transactiont0 t1 t2 t3 t4 t5t0 : Master has obtained control and asserts address, direction (not read), data. Waits a specified amount of time for slaves to decode targett1: Master asserts request linet2: Slave asserts ack, indicating data receivedt3: Master releases reqt4: Slave releases ackAsynchronous Handshake (4-phase)BussesThis is Fig. 6.11Chap. 6 - Storage*AddressDataReadReqAckMaster Asserts AddressNext Addresst0 t1 t2 t3 t4 t5t0 : Master has obtained control and asserts address, direction, data Waits a specified amount of time for slaves to decode target\t1: Master asserts request linet2: Slave asserts ack, indicating ready to transmit datat3: Master releases req, data receivedt4: Slave releases ackRead TransactionSlave DataBussesChap. 6 - Storage*All signals sampled on rising edgeCentralized Parallel Arbitrationoverlapped with previous transactionAll transfers are (unlimited) burstsAddress phase starts by asserting FRAME#Next cycle “initiator” asserts cmd and addressData transfers happen on whenIRDY# asserted by master when ready to transfer dataTRDY# asserted by target when ready to transfer datatransfer when both asserted on rising edgeFRAME# de-asserted when master intends to complete only one more data transferEXAMPLE: PCI Read/Write TransactionsBussesChap. 6 - Storage*– Turn-around cycle on any signal driven by more than one agentEXAMPLE: PCI Read TransactionBussesChap. 6 - Storage*How The CPU Talks To The IOThe interface consists of setting up the device with what operation is to be performed-Read or WriteSize of transferLocation on deviceLocation in memoryThen triggering the device to start the operationWhen operation complete, the device will interrupt.Interfacing I/O To The ProcessorI/O instructions (in,out) unique from memory access instructions.LDD R0,D,P 0 .1 x 10-6 sec/byte => 0.1 µsec/byte => 1000 bytes = 100 µsec 1000 transfers x 100 µsecs = 100 ms = 0.1 CPU secondsUser program progress only halted during actual transfer. Interrupt handler code does the transfer.1000 transfers at 1000 bytes each: 1000 interrupts @ 2 µsec per interrupt 1000 interrupt service @ 98 µsec each = 0.1 CPU secondsStill far from device transfer rate! 1/2 in interrupt overheadInterfacing I/O To The ProcessorChap. 6 - Storage*Delegating I/O Responsibility from the CPU: DMADirect Memory Access (DMA):External to the CPUAct as a master on the busTransfers blocks of data to or from memory without CPU interventionCPUIOCdeviceMemoryCPU sends a starting address, direction, and length count to IOC. Then issues "start".IOC provides handshakesignals for PeripheralController, and MemoryAddresses and handshakesignals for Memory.Interfacing I/O To The ProcessorChap. 6 - Storage*Transfer Method 3:Direct Memory AccessCPUIOCdeviceMemoryTime to do 1000 xfers at 1000 bytes each:1 DMA set-up sequence @ 50 µsec1 interrupt @ 2 µsec1 interrupt service sequence @ 48 µsec.0001 second of CPU timeCPU sends a starting address, direction, and length count to DMAC. Then issues "start".IOC provides handshake signals for PeripheralController, and Memory Addresses and handshakesignals for Memory.0ROMRAMPeripheralsIO BuffersnMemory Mapped I/OInterfacing I/O To The ProcessorChap. 6 - Storage*RAIDRedundant Array of Independent DisksIn this section we will:Motivate a need to have greater reliability and availability for disk data.Look at ways to get this greater reliability.6.1 Introduction6.2 Types of Storage Devices6.3 Busses - Connecting IO Devices to CPU/Memory. Interrupts etc. How is data transferred.6.5 Reliability, Availability and RAIDChap. 6 - Storage*Array Reliability Reliability of N disks = Reliability of 1 Disk ÷ N 1,200,000 Hours ÷ 100 disks = 12,000 hours 1 year = 365 * 24 = 8700 hours Disk system MTTF: Drops from 140 years to about 1.5 years!• Arrays (without redundancy) too unreliable to be useful!Hot spares support reconstruction in parallel with access: very high media availability can be achievedRAIDChap. 6 - Storage*Redundant Arrays of Disks• Files are "striped" across multiple spindles• Redundancy yields high data availabilityDisks will failContents reconstructed from data redundantly stored in the arrayCapacity penalty to store itBandwidth penalty to updateMirroring/Shadowing (high capacity cost)ParityTechniques:RAIDChap. 6 - Storage*Redundant Arrays of DisksRAID 1: Disk Mirroring/Shadowing• Each disk is fully duplicated onto its "shadow" Very high availability can be achieved• Bandwidth sacrifice on write: Logical write = two physical writes• Reads may be optimized• Most expensive solution: 100% capacity overheadTargeted for high I/O rate , high availability environmentsrecoverygroupRAIDProbabliity of failure (assuming 24 hours MTTR) = 24 / ( 1.2 X 106 X 1.2 X 106 ) = 6.9 x 10-13 = 170,000,000 yearsChap. 6 - Storage*Redundant Arrays of Disks RAID 3: Parity DiskP100100111100110110010011. . .logical record10010011110011011001001100110000Striped physicalrecords• Parity computed across recovery group to protect against  hard disk failures 33% capacity cost for parity in this configuration wider arrays reduce capacity costs, decrease expected availability, increase reconstruction time• Arms logically synchronized, spindles rotationally synchronized logically a single high capacity, high transfer rate diskTargeted for high bandwidth applications: Scientific, Image ProcessingRAIDChap. 6 - Storage*Redundant Arrays of Disks RAID 5+: High I/O Rate ParityA logical writebecomes fourphysical I/OsIndependent writespossible because ofinterleaved parityReed-SolomonCodes ("Q") forprotection duringreconstructionD0D1D2D3PD4D5D6PD7D8D9PD10D11D12PD13D14D15PD16D17D18D19D20D21D22D23P...............Disk ColumnsIncreasingLogicalDisk AddressesStripeStripeUnitTargeted for mixedapplicationsRAIDChap. 6 - Storage*Problems of Disk Arrays: Small WritesD0D1D2D3PD0'++D0'D1D2D3P'newdataolddataold parityXORXOR(1. Read)(2. Read)(3. Write)(4. Write)RAID-5: Small Write Algorithm1 Logical Write = 2 Physical Reads + 2 Physical WritesRAIDChap. 6 - Storage*Subsystem Organizationhostarraycontrollersingle boarddisk controllersingle boarddisk controllersingle boarddisk controllersingle boarddisk controllerhostadaptermanages interfaceto host, DMAcontrol, buffering,parity logicphysical devicecontrolstriping software off-loaded from host to array controllerno applications modificationsno reduction of host performanceRAIDCacheChap. 6 - Storage*System Availability: Orthogonal RAIDsArrayControllerStringControllerStringControllerStringControllerStringControllerStringControllerStringController. . .. . .. . .. . .. . .. . .Data Recovery Group: unit of data redundancyRedundant Support Components: fans, power supplies, controller, cablesEnd to End Data Integrity: internal parity protected data pathsRAIDChap. 6 - Storage*System-Level AvailabilityFully dual redundantI/O ControllerI/O ControllerCache &Array ControllerCache & Array Controller. . .. . .. . .. . .. . ....RecoveryGroupGoal: No SinglePoints ofFailurehosthostwith duplicated paths, higher performance can beobtained when there are no failuresRAIDChap. 6 - Storage*Summary6.1 Introduction6.2 Types of Storage Devices6.3 Busses - Connecting IO Devices to CPU/Memory. Interrupts etc. How is data transferred.6.5 Reliability, Availability and RAIDChap. 6 - Storage*Course SummaryDuring this course, we’ve started to learn about the details of computer architecture. Items included:Instruction Sets - especially a glimpse at the Intel instruction set.Pipelines - the gyrations necessary to speed up the processor.Memory - the various elements in the hierarchy designed to speed up the effective access to data.IO - a brief look at disks, busses, and how they are put together.