Have you ever thought about a question:
An alaccelerator worth hundreds of thousands of dollars, a large model that runs hundreds of billions of parameters, and suddenly the grid shakes, and it shakes less than 50 milliseconds — what is the result
It's not "the image is stuck." it's a training schedule that goes straight to zero, three days dry。
It's not alarmist. One cloud manufacturer in the country released a set of data last year: the disruption of the task of the training cluster due to the quality of the electricity supply caused an average loss of about 17 hours of effective training time each time. The cost of calculating a kilocalorie was interrupted once, sufficient to buy a luxurious medium sedan。
From this perspective, we're starting today to talk about how much the ai chip is “picking electricity” and why an invisible hal closed cyclops sensor will be a key node of the last line of defence in this chain。
I. The tolerance of the ai chip to voltage is much harder than you think
First of all, there is the notion that in the power industry it is called "voltage band tolerance"。
Ordinary household electricity, such as air conditioners, refrigerators, voltage of electricity floats at a nominal value of 10 per cent, with little impact on use. Server power is higher and dynamic response is ms. At the level of ai training chips, things are completely different。
For example, an ai training card that is currently in the mainstream market:
Core work voltage is only 0. 8 to 1. 2v, some of which are even lower
The single-card current can run up to thousands of annes at once
The voltage texture must be contained within 1%, and the precision calculation unit requires even 0. 5%
The load jumps from 10% to 100%. The charge drops back in microseconds
To convert it, the 1v power supply, the 1% texture tolerance limit, is 10 mv. What concept? The amount of voltage that you can generate when your cell phone is in poor contact with the wire can be greater。
That's why we say that the ai chip has a “voltage-cleaning habit” — a “electric tiger” with huge appetites but extreme demands for food quality. There was a slight lack of appetite, a minor error of calculation, a silent breakdown of the data, and a heavy chip that directly triggers a protective shutdown。
What's the worst thing about a big model trainee? It's not expensive, it's training until day 15, because all checkpoints are useless because of a flash。
Ii. Ups is not "already powered," but "right powered."
A lot of people know that ups is a big charge -- the market's out, it can top it。
That's right, but not all。
For the ai data centre, the real value of ups is not “electricity”, but “pure power”. Because the wave shape of the city's electricity grid is already dirty, there's a lot of waves, waves and transients. What ups does is filter these dirty things out and export a clean swirl to the back of the device。
How does ups work inside? In short, three steps:
Communication input, whirlwind flow, whirlwind, reverse into clean communication output
The hardest part of this is "reverse." the reversers need to know in real time: what are the currents being exported? Is the wave shape right? Is there any malformation? The load in the back suddenly pulled up, did i keep up
Who gave it that information。
The sensor retrieves the current signal and sends it to the controller, which adjusts the steering time for the switch. The whole set of closed loops is running down, and it's going to take dozens of microseconds a cycle, and it's going to be hundreds of microseconds。
The question arises: how do you adjust the controller if the sensors don't pick up the signal, delay, and float at any higher temperatures
The sensors can't see
The error of each ring in this chain is magnified laterally, and eventually hits the priceless computing hardware. So i say the current sensor is the nerve end of ups -- it does not output power, but it determines the quality of power output。
Iii. Why does it have to be closed
There are a number of current detection programmes, with separate, open-ring, closed-ring scenarios. But at the reverse output end of the high ups, there's basically a consensus in the circle that the closed circle hall is the first choice。
Why? Directly above:
Where's the ringhead for hall? Its accuracy is limited by the b-h curve of the magnetic core material. Once the temperature goes up, the magnetic core features change, the output signal floats. Also, non-linear errors in large currents are evident, with a number of compensatory algorithms to be made in control software, which may not be easy。
There's a difference of thought about this. It uses the principle of magnetic balance, also known as the principle of zero magnetic flux:
Primary currents generate a magnetic field, and secondary loops go up to reverse currents, creating a magnetic field that just offsets the magnetic field on the side. The hole component does not directly measure the size of the magnetic field, but rather the “no magnetic field”. Secondary currents are comparable to primary currents, and accuracy is determined by the number ratio and sample resistance, with little relation to the non-linear nature of magnetic cores。
There are two advantages to doing so:
One is that precision does not depend on magnetic cores. Magnetic cores always work near the zero-magnetic point, and the troubles with the b-h curve are largely bypassed. The linearity is good, with a full-scale accuracy of 0. 3% and 0. 5%。
The second is that warm and natural. Because the principle of work determines that it is not sensitive to temperature and does not require additional temperature compensation as is the case with the ring. The ai data centre is 7x24 hours full, the ups cabinet is home-to-house at 45 degrees per year, and cold drift means running down the year, with sampling signals largely neutral。
And one thing is easy to ignore: closed loops respond quickly, and bandwidth can be 200 khz. Large models are trained with extremely high load fluctuations, instant current jumps are large and fast, and sensors cannot keep up, directly determining whether the controller can react in a timely manner。
How does the current sensor match this scenario
Talk back to the current sensor. One of the sensors in the country has multiple closed-ring hole sensors that match the height of the scene:
The cm-a system consists of several series: range 100a to 2000a, with as much as ±0. 3% accuracy. Inverted output of mainly high power ups, straight-stream screen master detection. The maximum measurement error at 1000a is only 3a. For ups, which requires precise wave-form control, this surplus is a luxury。
Cr-a series: range 50a to 300a, accuracy > 0. 5%. Suitable for small and medium power ups, modularized power. Cost is more favourable, but precision is sufficient in this power segment。
Choose which one, see your ups power level and cost budget. But the core logic remains the same: the power supply at the ai data centre should not be refined on the sensor material。
We did a comparison test at the lab: the same ups platform, running in the same greenhouse with open and closed programs, rising from constant temperature to 55°c, the output deviation of the open ring became visible, and the basic texture of the closed ring remained intact. That is the difference that the rationale determines, not the adjustment of parameters。
V. Writing to the end
There is a widespread saying in the industry that the end of ai is power, the end is power。
I'd like to add at the end: the end of the power is precision detection and control。
Most of the discussion was about the infrastructure of computing, and the topics revolve around chips, light modules and liquid cooling. But the engineers who actually do the first line of work know that most of the time, the reason why the whole cluster is " hanging off" is not because the chip is burned, not because the network is broken, but because of some poor power supply. – for example, a current sampling signal from a ups drifted, causing an output wave malformation and triggering the protection of downstream equipment。
Such failures are the most difficult to detect and the most vulnerable to neglect。
So, the next time you see a big model, you're going to do another amazing round of training, think about the sensors that sit in the cabinet and stare at current fluctuations 24 hours a day. They don't work, but without them, they don't work。
That's what precision tests mean -- you can't see, but you can't。
