The Evolution of the V2Ray Protocol Stack: Technical Integration and Security Considerations from VMess to VLESS and XTLS

As a powerful network proxy tool, the evolution of V2Ray's core protocol stack profoundly reflects the relentless pursuit of performance, security, and usability. From the classic VMess to the lightweight VLESS, and then to the deep integration with XTLS, each iteration aims to address pain points in specific scenarios. Understanding this evolutionary path is crucial for building and optimizing proxy networks.

Phase One: The VMess Protocol - The Foundation of Security and Functionality

VMess (Versatile Messaging over Secure Stream) was the core transport protocol designed in the early stages of the V2Ray project. Its design goals were to provide strong encryption and authentication while maintaining sufficient flexibility and anti-detection capabilities.

Core Features and Design

1. Strong Authentication Mechanism: Uses Time-based One-Time Password and UUID (User ID) for mutual authentication between client and server, effectively preventing replay attacks and unauthorized connections.
2. Command-Response Structure: Communication begins with a Command, the server verifies and responds, and then data transmission proceeds. This structure adds complexity to the protocol state but provides stronger control capabilities.
3. Built-in Encryption: The protocol itself integrates an encryption layer (e.g., AES-128-GCM), ensuring the confidentiality and integrity of transmitted data.

Advantages and Limitations

• Advantages: High security, complete functionality, long-term battle-tested, wide community support.
• Limitations: Relatively large protocol header, performance overhead from the encryption/decryption process, relatively complex protocol structure.

Phase Two: The VLESS Protocol - Pursuing Minimalism and Efficiency

VLESS (Vision-LESS) emerged as a simplified successor to VMess. Its core design philosophy is "removing the unnecessary," stripping away non-essential features to pursue higher performance and a cleaner architecture.

Core Improvements

1. Protocol Slimming: Removed the built-in encryption layer present in VMess. VLESS itself does not handle encryption, delegating encryption entirely to the underlying transport layer (e.g., TLS). This makes the protocol itself extremely lightweight, reducing computational overhead.
2. Structural Simplification: Simplified the handshake process and protocol header format, reducing implementation complexity and parsing overhead.
3. UUID-Only Authentication: Retained UUID as the core authentication mechanism, but with a more direct process.

Impact

• Performance Improvement: Due to its lightweight nature, when paired with an efficient transport layer (like XTLS), it can significantly increase throughput and reduce latency.
• Separation of Concerns: Adheres to the "single responsibility" principle. VLESS focuses on proxying and routing, while encryption is handled by a dedicated transport layer, resulting in a clearer architecture.
• Deployment Requirements: Since it has no built-in encryption, VLESS must be used in conjunction with an encrypted transport layer like TLS, placing higher demands on server configuration.

Phase Three: The Introduction of XTLS - A Transport Layer Revolution

XTLS (Xray Transport Layer Security) is not an independent proxy protocol but a deep optimization and extension of the standard TLS transport layer. It was initially designed for the VLESS protocol to address performance bottlenecks in TLS proxying.

Technical Principles and Integration

1. "Flow Reuse" Mode: This is the core innovation of XTLS. In standard TLS proxying, data undergoes "client decryption -> proxy core processing -> server encryption" or the reverse process, i.e., multiple rounds of encryption/decryption. XTLS modes like direct allow, after successful authentication, the direct forwarding of part or all of the original TLS data stream, avoiding the proxy server's repeated encryption/decryption of the data content.
2. Deep Integration with VLESS: The lightweight nature of VLESS makes it the perfect partner for XTLS. VLESS handles the proxy logic and initial authentication. Once passed, XTLS can take over subsequent data transmission, achieving efficiency close to that of a native TLS connection.
3. Security Boundary: XTLS's "flow reuse" is built upon strict identity authentication (performed by VLESS) and TLS handshake verification. It optimizes the efficiency of data transmission after authentication, not by weakening the security of the authentication itself.

Security Considerations and Protocol Selection

Throughout the evolution of the protocol stack, security remains the primary consideration.

1. VMess vs. VLESS+XTLS Security:

   • VMess provides "full-stack" security, self-contained from authentication to encryption.
   • VLESS+XTLS decomposes security responsibilities: VLESS handles strong authentication, XTLS (based on standard TLS) handles transport encryption. As long as TLS is configured properly (using strong cipher suites, valid certificates), its security is equally high. The key lies in the strength of the TLS layer configuration.

2. Anti-Detection Capability:

   • Both VMess and VLESS over TLS have good camouflage capabilities, with traffic characteristics similar to ordinary HTTPS connections.
   • The traffic characteristics of XTLS's direct mode are closer to a real TLS connection, but theoretically, any deep interference with TLS traffic could affect its special modes. It is generally recommended to use more balanced modes like xtls-rprx-vision.

3. Practical Recommendations:

   • Pursuing Ultimate Performance & Modern Deployment: First choice is VLESS + XTLS (vision) + TLS 1.3. This is currently one of the most performant configurations, suitable for high-bandwidth, low-latency scenarios.
   • Requiring Broad Compatibility & Robustness: VMess over TLS remains a reliable choice with wider client support.
   • Security First: Regardless of the chosen protocol, transport layer encryption (TLS) must be enabled, using valid certificates (recommended ACME auto-issued), and insecure encryption methods must be disabled.

Conclusion

The evolution of the V2Ray protocol stack from VMess to VLESS+XTLS is a path from "integrated functionality" to "separation of concerns and deep optimization." VLESS improves efficiency by simplifying the protocol core, while XTLS breaks performance bottlenecks by innovating the transport layer. This combination provides modern proxy networks with near-native transmission performance while maintaining a high degree of security and flexibility. Understanding the technical characteristics and applicable scenarios of these protocols helps us make the most reasonable technology selections and configurations based on actual needs.