close search bar

Sorry, not available in this language yet

close language selection

Standard versus proprietary security protocols

Chandu Ketkar

May 28, 2014 / 3 min read

What are standard security protocols?

The encyclopedia defines a security protocol as “a sequence of operations that ensure protection of data. Used with an underlying communication protocol, it provides secure delivery of data between two parties.”

We use security protocols in everyday computing. For example, when we use our domain credentials to log in to a Microsoft Windows environment, we use the Kerberos (or the NTLM protocol in the earlier versions of Windows) indirect authentication protocol. When we conduct an e-commerce transaction using a secure e-commerce site, we use various encryption protocols, integrity protocols, and so on. Of course, there are other types of security protocols – authentication protocols, access-control (or authorization protocols), encryption protocols, key management protocols, key distribution protocols, and so on.

In day-to-day computing, we use standard security protocols because they offer us significant security benefits.

For one, experts design standard security protocols. Once designed, these are reviewed and re-reviewed by experts in standards organizations (e.g., IEEE, W3C, IETF). New protocols are subjected to security threat modeling analysis to ensure that they offer protection against commonly known attack patterns. When these protocols are deployed in the field, their security is monitored, and over time their security kinks are worked out.

In addition, when these standard protocols become (eventually) insecure, more secure versions of protocols are made available (e.g., TLS 1.1/1.2 to replace TLS 1.0), or new protocols are designed to replace the aging ones (e.g., AES replaced aging DES/3DES).

There are weaknesses associated with the use of standard protocols as well. Since all the knowledge pertaining to a standard protocol design/implementation is in the public domain, there is a greater likelihood of finding zero-day exploits. Additionally, when an issue discovered, it is widely known and exploitable in a very short period of time. The Heartbleed bug is a good example of this. Finally, tools are widely available to exploit any older and unpatched versions of the software libraries/packages that are known to implement the standard protocol incorrectly.

Why use proprietary security protocols?

In the world of embedded devices, there are a few reasons why people may choose to use proprietary protocols:

Security through obscurity

  • A proprietary protocol by definition is obscure, and an attacker will most likely need time and resources (usual exploiting tools will not work as an example) to understand how the protocol is designed and implemented before exploiting any weaknesses.
  • Proprietary protocols can help you avoid a scenario where software updates are required (which are expensive for embedded devices) because a vulnerability is found in a standard protocol in a different context. If standard protocols are implemented, then with each known issue, a software update will be required. This can be an expensive proposition. With proprietary protocols, we are safe – at least until someone goes after our device specifically.


Most embedded devices have limited computing power. In such cases, there is a need or tendency to use lightweight / proprietary protocols.

What are some issues with proprietary standard protocols?

Whatever the reasons are, when we design our own proprietary protocols, we need to worry about many security issues.

For example, if we are designing a custom cryptographic protocol (a really really bad idea), then we need to worry about whether our protocol is provably secure. If not, then is our proprietary protocol secure against passive attacks (CPA-secure) or active attacks (CCA-secure)?

If we are designing a distributed proprietary protocol (say, an custom authentication protocol or a custom key distribution protocol), then we need to worry many other security issues, such as these:

  • Is the proprietary protocol vulnerable to man-in-the-middle (MitM) attacks? Can an attacker insert himself/herself in the middle to breach the protocol?
  • Is our proprietary protocol vulnerable to replay attacks? Can the protocol packets be captured and replayed at a later point to confuse either endpoint?
  • Can an attacker find other ways to spoof the endpoints?
  • Can an attacker launch an interleaving attack to break the proprietary protocol?
  • We may even need to worry about reflection attacks, forced-delay attacks, and the list goes on.

Should I use a standard security protocol or create my own?

At Synopsys, we conduct reviews of thick clients / embedded systems / mobile applications in the healthcare, gaming, and financial industries, among others. In our reviews, we commonly find serious design-level and implementation-level issues in proprietary security protocol implementations. We've observed that it's very difficult to design and implement a proprietary protocol correctly. Our security analysts are routinely able to find serious vulnerabilities in such protocols. A common example of such proprietary security protocol code is X.509 certificate validation. While most commonly used browsers do a good job of validating server-side X.509 certificates, we routinely find many issues when mobile applications (which are responsible for certificate validation) implement such validation.

Our recommendation is to use standard security protocols as much as possible. If you must use a proprietary security protocol, try to use it in conjunction with a standard protocol (e.g., implementing proprietary encryption or encoding scheme over a secure TLS link, or using custom white box cryptography controls in conjunction with other well-known obfuscation controls). If you use a proprietary security protocol exclusively, you must thoroughly assess its design and implementation for security vulnerabilities.

Continue Reading

Explore Topics