Message Queue

Definition

In computer science, a message queue is a method of inter-process communication or communication between different threads of the same process. A software queue is used to process a series of input, usually from the user.

The message queue provides an asynchronous communication protocol. Each record in the storage column contains detailed data, including the time of occurrence, the type of input device, and specific input parameters. That is to say: the sender and receiver of the message do not need to interact with the message queue at the same time. The message remains in the queue until the recipient retrieves it.

Usage scenarios

When the results do not need to be obtained immediately, but the amount of concurrency needs to be controlled, it is almost time to use the message queue.

Message queue mainly solves problems such as application coupling, asynchronous processing, and traffic cutting.

Common message queue

Currently, the most commonly used message queues include RabbitMQ, RocketMQ, ActiveMQ, Kafka, ZeroMQ, MetaMq, etc., and some databases such as Redis, Mysql and phxsql can also implement message queue functions.

Advantages and Disadvantages

The message queue itself is asynchronous, which allows the receiver to retrieve the message long after the message is sent, which is different from most communication protocols. For example, the HTTP protocol used in the WWW (before HTTP/2) was synchronous because the client had to wait for the server to respond after making a request. However, there are many situations where we need asynchronous communication protocols. For example, one process notifies another process that an event has occurred, but does not need to wait for a response. However, the asynchronous nature of the message queue also creates a disadvantage, that is, the receiver must poll the message queue to receive the latest message.

Compared with signals, message queues can transmit more information. Compared with pipes, message queues provide formatted data, which can reduce the developer’s workload. But message queues still have size limits.

In addition to being a buffer between different threads or processes, the message queue can also detect whether there are performance problems with the receiving thread or process through the current number of messages in the message queue.

Message queue usage scenarios

Application coupling: multiple applications process the same message through the message queue to avoid failure of the entire process due to interface call failure;
Asynchronous processing: Multiple applications process the same message in the message queue, and messages are processed concurrently between applications. Compared with serial processing, processing time is reduced;
Current limiting and peak shaving: widely used in flash sales or rush buying activities to avoid application system hang-ups caused by excessive traffic;
Message-driven system: The system is divided into message queue, message producer, and message consumer. The producer is responsible for generating messages, and the consumers (there may be multiple) are responsible for processing messages;

Application coupling

Specific scenario: The user uploads a picture using the QQ album, and the face recognition system will perform face recognition on the picture. The general approach is that after the server receives the picture, the picture upload system immediately calls the face recognition system, and then returns success after the call is completed.

This method has the following disadvantages:

The face recognition system failed to be adjusted, causing the image upload to fail;
High latency, requiring the face recognition system to complete processing before returning it to the client, even if the user does not need to know the result immediately;
The picture upload system and the face recognition system call each other and need to be coupled;

If using message queue:

After the client uploads the image, the image upload system writes the image information into the message queue and directly returns success; while the face recognition system regularly fetches data from the message queue to complete the recognition of the newly added images.

At this time, the image upload system does not need to care whether or when the face recognition system processes the image information. In fact, since users do not need to know the face recognition results immediately, the face recognition system can choose different scheduling strategies to process the picture information in the queue according to idle time, busy time, and normal time.

Asynchronous processing

Specific scenario: In order for a user to register to use an application, the system needs to send a registration email and verify the SMS. There are two ways to handle these two operations: serial and parallel.

Serial method: After the new registration information is generated, the registration email is sent first, and then the verification SMS is sent;
Parallel processing: After the new registration information is written, it is processed in parallel by sending text messages and sending emails;

In serial mode, you need to send the registration email first and then send the verification SMS and then return it to the client. Parallel processing, text messaging and email processing are completed before being returned to the client.

Assuming that the three steps of generating registration information, sending registration emails, and sending verification text messages all take 50ms, then the processing time of the above two methods is:

Serial: 50+50+50=150ms Parallel: 50+50 = 100ms

If using message queue:

If success is returned to the client immediately after writing to the message queue, the total response time depends on the time of writing to the message queue. The time of writing to the message queue itself can be very fast and can be basically ignored. Therefore, the total processing time is doubled compared to serial and doubled compared to parallel;

The premise is that the registration email and verification SMS do not need to be real-time.

Current limiting and peak shaving

Specific scenario: A shopping website carries out a flash sale activity. Generally, due to excessive instantaneous visits and excessive server reception, traffic will surge, and the relevant system will be unable to process requests or even crash. After joining the message queue, the system can fetch data from the message queue, which is equivalent to a buffering of the message queue.

This method has the following advantages:

The request is put into the message queue first, instead of being processed directly by the business processing system, and a buffer is made, which greatly reduces the pressure on the business processing system;
The queue length can be limited. In fact, during the flash sale, users who enter the queue later cannot get the product. These requests can be directly abandoned and the information that the event has ended or the product has been sold out will be returned;

Message-driven system

Specific scenario: The user has uploaded a new batch of photos. The face recognition system needs to cluster all the photos of this user. After the clustering is completed, the reconciliation system will regenerate the user’s face index (to speed up the query). These three subsystems are connected by message queues. The processing results of the previous stage are put into the queue, and the latter stage obtains messages from the queue to continue processing.

This method has the following advantages:

Avoid directly calling the next system causing the current system to fail;
Each subsystem can be more flexible in how to process messages. You can choose to process messages when they are received, you can choose to process them regularly, or you can divide time periods and process them at different processing speeds;

Two modes of message queue

Message queue includes two modes, point-to-point mode (point to point, queue) and publish/subscribe mode (publish/subscribe, topic).

Peer-to-peer mode

Peer-to-peer mode includes three characters:

Message queue
sender (producer)
Receiver (consumer)

The message sender produces the message and sends it to the queue, and then the message receiver takes the message from the queue and consumes the message. After the message is consumed, there is no longer storage in the queue, so the message receiver cannot consume the message that has been consumed.

Point-to-point mode features:

Each message has only one receiver (Consumer) (that is, once consumed, the message is no longer in the message queue);
There is no dependency between the sender and the receiver. After the sender sends a message, whether the receiver is running or not, it will not affect the sender’s next message sending;
After successfully receiving the message, the receiver needs to respond successfully to the queue so that the message queue can delete the currently received message;

Publish/subscribe model

Character theme (Topic)
Publisher
Subscriber

The publisher sends messages to the Topic, and the system delivers these messages to multiple subscribers.

Features of publish/subscribe model:

Each message can have multiple subscribers;
There is a time dependency between publishers and subscribers. For a subscriber of a certain topic (Topic), it must create a subscriber before it can consume the publisher’s messages.
In order to consume messages, subscribers need to subscribe to the role topic in advance and keep it running online;

Introduction to common message queues

This section mainly introduces the main features, advantages, and disadvantages of four commonly used message queues (RabbitMQ/ActiveMQ/RocketMQ/Kafka).

The following information may be inconsistent with some descriptions due to software development iterations, that is, the following introduction, advantages and disadvantages may be out of date, and the collection time is around the end of 2017.

RabbitMQ

Released in 2007, RabbitMQ is a reusable enterprise messaging system based on AMQP (Advanced Message Queuing Protocol). It is one of the most mainstream message middleware currently.

Main features:

Reliability: Provides a variety of technologies that allow you to trade off performance and reliability. These technologies include persistence mechanisms, delivery confirmations, publisher attestations, and high availability mechanisms;
Flexible routing: Messages are routed through switches before reaching the queue. RabbitMQ provides several built-in switch types for typical routing logic. If you have more complex routing requirements, you can combine these switches. You can even implement your own switch type and use it as a RabbitMQ plug-in;
Message clustering: Multiple RabbitMQ servers in the same LAN can be aggregated together and used as an independent logical agent;
Queue high availability: Queues can be mirrored on machines in the cluster to ensure message security despite hardware problems;
Support for multiple protocols: Supports multiple message queue protocols;
The server side is written in Erlang language and supports all programming languages you can think of;
Management interface: RabbitMQ has an easy-to-use user interface that allows users to monitor and manage many aspects of the message broker;
Tracking mechanism: If the message is abnormal, RabbitMQ provides a message tracking mechanism so that users can find out what happened;
Plug-in mechanism: Many plug-ins are provided to extend it in many aspects, and you can also write your own plug-ins;

Advantages:

Due to the characteristics of the Erlang language, mq has better performance and high concurrency;
Robust, stable, easy to use, cross-platform, supports multiple languages, and has complete documentation;
There is a message confirmation mechanism and persistence mechanism, with high reliability;
Highly customizable routing;
The management interface is rich and has large-scale applications in Internet companies;
High community activity;

Disadvantages:

Although combined with the concurrency advantages of the Erlang language itself, the performance is better, but it is not conducive to secondary development and maintenance;
Implemented a broker architecture, meaning messages can be queued on a central node before being sent to the client. This feature makes RabbitMQ easy to use and deploy, but makes it run slower because the central node increases latency and the message encapsulation is larger;
Need to learn more complex interfaces and protocols, and the cost of learning and maintenance is high;

ActiveMQ

ActiveMQ is produced by Apache. ActiveMQ is a JMS Provider implementation that fully supports JMS1.1 and J2EE 1.4 specifications. It is very fast, supports multiple language clients and protocols, can be easily embedded into enterprise application environments, and has many advanced features.

Comply with the JMS specification: The JMS specification provides good standards and guarantees, including: synchronous or asynchronous message distribution, one-time and only-once message distribution, message reception and subscription, etc. The benefit of conforming to the JMS specification is that these basic features are available regardless of the JMS implementation provider used;
Connectivity: ActiveMQ provides a wide range of connection options, supported protocols are: HTTP/S, IP multicast, SSL, STOMP, TCP, UDP, XMPP, etc. Support for many protocols gives ActiveMQ great flexibility.
Supports many protocols: OpenWire, STOMP, REST, XMPP, AMQP;
Persistence plugins and security plugins: ActiveMQ provides a variety of persistence options. Moreover, ActiveMQ’s security can also be fully customized for authentication and authorization based on user needs;
Supports many types of client languages: in addition to Java, there are also: C/C++, .NET, Perl, PHP, Python, and Ruby;
Broker cluster: Multiple ActiveMQ brokers can form a cluster to provide services;
Extraordinarily simple management: ActiveMQ is designed with developers in mind. Therefore, it does not require a dedicated administrator as it provides simple and easy-to-use management features. There are many ways to monitor data at different levels of ActiveMQ, including using JMX in JConsole or ActiveMQ Web Console, by processing JMX alarm messages, by using command line scripts, and even by monitoring various types of logs.

Advantages:

Cross-platform (JAVA writing has nothing to do with the platform, ActiveMQ can run on almost any JVM)
JDBC can be used: data can be persisted to the database. Although using JDBC will reduce the performance of ActiveMQ, the database has always been the storage medium most familiar to developers. Save the message to the database and make it visible and tangible. Moreover, the company has a dedicated DBA to tune the database and separate the master from the slave; -Support JMS: Support the unified interface of JMS; -Support automatic reconnection;
Security mechanism: supports multiple security configuration mechanisms based on shiro, jaas, etc., and can authenticate and authorize Queue/Topic.
Complete monitoring: It has complete monitoring, including Web Console, JMX, Shell command line, and Jolokia’s REST API;
Friendly interface: The provided Web Console can meet most situations, and there are many third-party components that can be used, such as hawtio;

Disadvantages:

Community activity is not as high as RabbitMQ;
According to feedback from other users, inexplicable problems will occur and messages will be lost;
The current focus is on activemq6.0 product-apollo, and there is less maintenance on 5.x;
Not suitable for application scenarios with thousands of queues;

RocketMQ

RocketMQ is an open source product from Alibaba and is implemented in the Java language. It referred to Kafka during design and made some of its own improvements. The message reliability is better than Kafka. RocketMQ is widely used in Alibaba Group in order, transaction, recharge, stream computing, message push, log streaming, binglog distribution and other scenarios.

Main features:

It is a queue model message middleware with high performance, high reliability, high real-time and distributed characteristics;
Producers, consumers, and queues can all be distributed;
Producer sends messages to some queues in turn, and the queue set is called Topic. If the Consumer does broadcast consumption, one consumer instance consumes all the queues corresponding to this Topic. If it does cluster consumption, multiple Consumer instances consume the queue set corresponding to this topic on average;
Able to ensure strict message order;
Provides rich message pulling modes;
Efficient subscriber horizontal expansion capabilities;
Real-time message subscription mechanism;
Billion-level message accumulation capability;
Less dependencies;

Advantages:

A single machine supports more than 10,000 persistent queues
All RocketMQ messages are persistent. They are first written to the system PAGECACHE and then flushed to the disk. This ensures that both the memory and the disk have a copy of the data. When accessed, the messages are read directly from the memory.
The model is simple and the interface is easy to use (JMS interface is not very practical in many situations);
The performance is very good, and a large number of messages can be accumulated in the broker;
Supports a variety of consumption, including cluster consumption, broadcast consumption, etc.
Distributed expansion design of each link, master-slave HA;
The development is relatively active and the version is updated quickly.

Disadvantages:

The RocketMQ community’s attention and maturity are not as good as the first two;
There is no web management interface, but a CLI (command line interface) management tool is provided to query, manage and diagnose various problems;
No interfaces such as JMS are implemented in the mq core;

Kafka

Apache Kafka is a distributed message publish and subscribe system. It was originally implemented by LinkedIn as a distributed commit log system (a distributed commit log) based on a unique design, and later became part of the Apache project. Kafka systems are fast, scalable, and durable. Its partitioning characteristics, replicability and fault tolerance are all good features.

Main features:

Fast persistence, message persistence can be carried out with O(1) system overhead;
High throughput, a throughput rate of 10W/s can be achieved on an ordinary server;
A complete distributed system, Broker, Producer, and Consumer all natively and automatically support distribution and automatically achieve load balancing;
Supports both synchronous and asynchronous replication HA;
Supports batch sending and pulling of data;
zero-copy: Reduce IO operation steps;
Data migration and expansion are transparent to users;
Machine expansion without downtime;
Other features: strict message order, rich message pull model, efficient subscriber horizontal expansion, real-time message subscription, billion-level message accumulation capability, regular deletion mechanism;

Advantages:

The client has rich languages, supporting java, .net, php, ruby, python, go and other languages;
Excellent performance, single-machine write TPS is about one million messages/second, message size is 10 bytes;
Provides a fully distributed architecture with a replica mechanism, which has high availability and reliability, and theoretically supports unlimited accumulation of messages;
Support batch operations;
Consumers use the Pull method to obtain messages, the messages are in order, and control can ensure that all messages are consumed and only consumed once;
There is an excellent third-party Kafka Web management interface Kafka-Manager;
Relatively mature in the field of logging, used by many companies and multiple open source projects;

Disadvantages:

If a single Kafka machine has more than 64 queues/partitions, the load will obviously skyrocket. The more queues, the higher the load, and the response time for sending messages will become longer.
Using short polling mode, real-time performance depends on the polling interval;
Retry is not supported if consumption fails;
Supports message order, but when one agent goes down, messages will be out of order;
Community updates are slow;

Conclusion:

Kafka lies in the distributed architecture, RabbitMQ is implemented based on the AMQP protocol, and RocketMQ/ideas are derived from kafka, changed to a master-slave structure, and optimized in terms of transaction reliability. Broadly speaking, those who have high transaction requirements such as e-commerce and finance can consider RabbitMQ and RocketMQ, and those who have high performance requirements can consider Kafka.

Summary:

The message queue uses an efficient and reliable message delivery mechanism for platform-independent data exchange, and integrates distributed systems based on data communication. There are currently many MQ products in the industry, such as RabbitMQ, RocketMQ, ActiveMQ, Kafka, ZeroMQ, MetaMq, etc. There are also cases where the database redis is directly used as a message queue. Each of these message queue products has its own focus. When making actual selection, you need to take comprehensive considerations into consideration based on your own needs and MQ product features.

Again, please note: The above software comparison information may be inconsistent with some descriptions due to software development iterations, that is, the software introduction, advantages and disadvantages may be out of date, and the collection time is around the end of 2017.

References

Introduction to message queues and common message queues RabbitMQ homepage ActiveMQ homepage RocketMQ homepage Kafka homepage