Summary
With the rapid growth of oif operations, the construction of traditional message-based intermediates faces the challenge of processing hundreds of millions of-grade equipment connections and big data. The paper analyses the structural trends and core strengths of the message medium and explores the practical applications of the apache pulsar on the china cloud iot platform, showing how the china cloud iot has built on it high-security, high-reliability, high-performance capabilities in supporting hundreds of-grade equipment and processing large-scale data streams。
Introduction to the online platform
China’s iotda service is a core base for china’s cloud networking, providing capacity for mass equipment to connect to clouds, equipment and cloud-end two-way messaging, data flow, equipment linkage rules, etc., as illustrated above. The data flow service is a core function of the iotda, which provides users with efficient and flexible data processing capabilities. When the equipment in the networked platform meets preset rule conditions, the platform automatically triggers the corresponding operation, helping users to achieve key business objectives, such as connecting data to china for other cloud services, and achieving data storage, computing and analysis throughout the stack. For a wide variety of users and developers, china plans multiple ways of receiving data for the cloud iotda。
Problems with extension of the property network platform
When it comes to the middle of the message, it is necessary to mention the fact that apache kafka, apache kafka, in the 2011 open source, has over the past period become almost a proxy for the middle of the message, a standard of fact on the message interface. In the meantime, coNfluent is also active in the commercialization of kafka and has also completed major features such as going to zookeeper, cascade storage in recent years. The platform initially used apache kafka as its message component。
Kafka, as an early runner, built up a wealth of users and rich scenery applications based on a simple concept, excellent performance. However, problems were revealed by the trend towards cloud application, rising business volume, and multi-lease of saas applications。
・info intermediate costs: because of reliability requirements, data are distributed among at least two azs in common high-availability schemes, the cost of which includes the cost of cloud disk storage, as well as the cost of some public cloud traffic across available areas. It is worth mentioning that some cloud manufacturers, such as huai yun, do not charge the cost of trans-usable area traffic。
・expanding time: more flexible cloud applications often require expansion at peaks of operations, kafka topic data is permanently on local disks, and once extended, data migration is required, which may reach hourly or skyscales。
・topic quantification limit: with the development of saas applications, news queue segregation among tenants becomes more important. Kafka's storage model, a partition corresponds to a disk file and relies on pagecache. The performance results were poor after the topic scale exceeded 5000。
・long tailings high: kafka messages need to wait for leader to be successful before they can be written, which is no advantage over the higher sla or quorum mode systems。
・consumer model single: message mediums need to support queue (shared) consumption patterns, thereby reducing the waste of resources when more consumers are connected. Operations like amqp, on the other hand, require this consumption pattern, otherwise they need to build their own capacity at the operational level。
Message media architecture trends and cloudy practice
In the face of these problems, market challengers such as automq, pulsar and warpstream have proposed their own solutions, which, according to the information available:
・reducing the cost of message intermediates: to reduce the cost of durability, it is possible to use object storage extensively (using cascade storage or direct storage or using disks only in wal). For cross-usable area traffic costs are unavoidable unless they are stored directly to a highly usable object。
・restrict the length of the extension: the length of the extension may require a reproduction of the content from the old disk due to the main reproduction mode from the disk, and at this point the writing continues, and the process may take longer to complete. This can be done by resolving the problem of storage expansion (using the object storage or the quorum model), or by minimizing what needs to be copied (cyclical storage to object storage)。
・extension of topic quantities: from the perspective of large data batch processing, the problem is not significant. However, there are strong single-channel claims in saas applications or on-line solutions, and the separation of equipment from equipment between tenants becomes more important. Usually addressed by the optimization of storage models, avoiding single topics against a single file handle。
・reducing long tail delay: it can be reduced by both wal and quorum techniques。
・diversification of consumption models: this characteristic is not a common problem, and the problem scene is rarely found in large (enterprise) intranet data processing when more consumers are less capable of sharing individual consumption. To date, only pulsar has supported the consumption of queue models, but kafka has also proposed kip-932 to support the consumption pattern, which is expected to be completed later。
The key points of divergence in technical decision-making are illustrated below in terms of broader strategies for achieving it (the dashed lines are optional modules)
・paint level, support one or more protocols, support the kafka agreement or support the pulsar agreement
Wal, own wal, will show a lower delay, but because the wal itself needs to be available, it is impossible to avoid a flow cost across the az
・store layer, whether to use object storage directly, or to store cold data offload to object storage
・stands, using disk storage, using main mode or quorum mode, quorum will have a more smooth delay (e. G. P99), but network popping may become complicated (usually for performance, the client will be connected directly to multiple broker)
According to the technical differences above, the technical routes of automq, kafka, pulsar and warpstream in the industry can be described as:
・automq: supports the kafka protocol, using wal, to permanently store messages in object storage。
・kafka: supports the kafka protocol (kafka of course supports the kafka protocol), does not have wal, permanently store it on the main disk from mode to object, while supporting the message middle of cascade to object storage。
・pulsar: supports multiple protocols such as kafka, pulsar, amqp, uses wal, permanently store on quorum 's disk, and supports the middle of messages linked to object storage。
. Warpstream: support the kafka protocol, no wal, permanent storage of messages stored in objects。
In general, there is a certain degree of consensus within the industry on the key optimization points of the message, and the agreement has to some extent created de facto standards (kafka agreement), which may become more important in detail design and engineering capabilities。
Apace pulsar was born early among kafka's challengers and, in a relatively comprehensive manner, better addressed the issues of extended time, topic quantum, long tails, and single consumption models. The final selection for the iotda is based on the core message middle of apache pulsar。
Apache pulsar
The apache pulsar, the top-level apache software foundation project, is the next generation cloud-based distributed streaming platform, which combines message, storage, light-quantification functions and uses computational and storage separation structures to support multi-tensor, durable, multi-chamber cross-regional data replication, with strong consistency, high-swallow, low-time and high-extension data storage features, and is considered to be the ideal platform for real-time stream transmission, storage and computing during the cloud lifetime。
In addition to using the cloud iot not only in depth, but also actively involved in the development of the pulsar community, working to enhance the safety, reliability and functionality of the pulsar, china has developed a number of enhancements, which are planned to be integrated into the community version in the future, in addition to the already consolidated 300+pr contribution (including downstream, peripheral projects)。
・debugging capacity enhancement: pulsar's own calibration capacity can only be observed by adjusting the log level of the broker to debug information, which is difficult to open in the production environment under a mass message scenario. The iot product department connects pulsar with the configuration centre, introduces a new configuration item, printing detailed debugging information for the specified theme only, thereby quickly identifying and resolving delays in messages from producer to service. When this feature goes online, it helps to sort out dozens of problems。
・performance-related enhancement: integration of skywalking 's call chain capability into pulsar and bookkeeper, providing detailed performance analysis. Helping users understand the performance of the system, identifying long-end requests and optimizing performance
・reliance is enhanced: the iot product department's strategy is to increase the system's self-rehabilitation capacity compared to the community's preference to solve problems after they occur. For example, when data on disks are detected to be inconsistent, automatically restarting to repair the data, where the failure rate of producers is too high, and when broker detects consumer anomalies (e. G., continuous production, acceptance line length but no consumption), the consumer link is automatically disconnected. While these measures may result in the loss of some malfunctioning location information, they compensate for the community's lack of automatic recovery and ensure that the system remains stable in exceptional circumstances。
Iot & futage technology case analysis
This technology was preceded by the centre for geo-information research and development (cgi) and is a subsidiary of the sunfung group. Work to create digital and ai-readable real-time accurate maps that provide locational intelligence decision-making services for enterprises and individuals。
Client pain:
1. Twenty times the number of simultaneous messages
2. High-frequency vehicle movements
The fusion-technology has 30,000 + cargo vehicle cameras for map filming, access to data flows up to 150,000 tps. The iot is a cost-saving platform for businesses, with examples of platforms that are highly usable, reliable and cost-effective, and supports demand-driven expansion。
China clouds iot with high-availability, high-reliability news bus, steady support for 150,000 tps data flows, quickly transferring data streams from the sky to high-level services such as ei, big data, and helping clients quickly to excavate data values. Through the trajectories of freight vehicles, it has acquired real-time street views, completed dynamic mapping, built new business models based on sensory data。
Summary
The active participation of the iot team in the community development of the apache pulsar and its downstream projects ensured that pulsar operated steadily for more than five years on the china cloud networking platform, thus enhancing the core competitiveness of the platform. In addition, the team has made significant contributions to other open-source projects such as apache bookkeeper, apolloconfig and opengemini. The department will continue to work actively with and support open-source communities, not only through technology-sharing and problem-solving, but also in the governance of open-source projects, the development of new functions and community-building, effectively promoting technological innovation and achieving synergy between iot operations and technology。
Through close cooperation with open-source communities, the high performance performance of the iot platform in a high and growing environment has been enhanced, providing reliable and efficient services to enterprises. The iot product team will continue to provide more comprehensive support for the efficient functioning of the goods network and the digital transformation。
