What is Secrets Management?

Photo by Richard Payette

What are application secrets?

Once upon a time, a website was just a static html page with some markup, and you might have gotten some styling if you were lucky. Although a few people still build websites this way, most modern applications are increasingly complex and rely on several pieces of interconnected infrastructure to work properly and deliver the experience that users expect.

Separate frontends, backends, databases, object storage buckets, CDNs, cache layers, and third-party APIs all need to talk to each other securely and reliably. Application secrets are the passwords, certificates, API tokens, and encryption keys that allow complex modern applications to work.

Why do we need to manage secrets?

Why are we talking about this? Application secrets pose unique challenges during the various stages of application development, deployment and management within an engineering team. There are ample opportunities for secrets to be mishandled, leading to disastrous consequences. Established developer-tooling like version-control or package management have no analog for managing secrets, and hacky work-arounds offer little by way of security and slow down development and deployment of code.

Here we attempt to give you the lay of the land of secrets management, outline some of the most common mistakes when handling secrets, and offer some best practices and tools to help you ship faster, without breaking things.

There’s a lot at stake

Leaked secrets are the most common cause of security breaches. Remember when Uber had to admit that hackers stole personal data of 57 million riders and drivers because they found credentials in a code repository? Unfortunately, there are countless more examples of leaked secrets and credentials that have caused millions of dollars in damage to companies and individuals.

Just as we are writing this post, Rabbit has come under fire for leaving hardcoded API keys in their R1 codebase, allowing anyone with access to view or alter all device responses (including personal info), and even completely brick all R1 devices in existence.

Developers deserve better

Working with secrets during development is painful. The lack of developer-focused secret management tools means secrets frequently end up being hardcoded or accidentally checked into git repositories. While using .env files is an improvement over hardcoding, they are far from a perfect solution: they lack a secure method for sharing secrets among team members, have no built-in access control, and store sensitive information in plaintext.

Most existing secret management tools, such as Hashicorp Vault or AWS Secrets Manager, are designed with large-scale deployments in mind, catering primarily to SREs and DevOps teams. These tools often don't fit well into the daily workflows of developers, making it difficult to adopt best practices for secret management during development. The result is a frustrating and inefficient experience that leaves security gaps and hinders productivity.

Cloud-provider solutions are silos

All major cloud providers offer secret management solutions, but they come with a significant drawback: vendor lock-in. Once you commit to a provider's secret management system, switching cloud providers becomes a complex and arduous task. Furthermore, deploying secrets to other platforms (Cloudflare Pages / Netlify for preview builds, or GitHub actions for a testing pipeline, for example), becomes nearly impossible, leading to secret sprawl. This lack of interoperability and flexibility is a feature rather than a bug of cloud-providers' secret managers.

Collaboration, access control & visibility

As an engineering team scales, it becomes increasingly necessary for secrets to be shared and accessible by all members of the team. Without a centralized secret management solution, developers often have to resort to insecure channels like Slack DMs or emails. Aside from the obvious security risk of leaking secrets through these methods, there's no way to track who has access to which secrets and how they're used. The end result is poor security practices and compromised developer productivity that gets worse as the team grows.

Common pitfalls in Secrets Management

With a better idea of why secret management matters, let's break down some of the common mistakes and bad practices that could be introducing vulnerabilities into your workflow and slowing down productivity.

Hardcoding secrets in code

import requests

# Hardcoded API key
API_KEY = '8796g3-jhg453-3h4g5jg3-mn4v4345'

def get_data():
    url = f'https://api.example.com/data?api_key={API_KEY}'
    response = requests.get(url)
    if response.status_code == 200:
        return response.json()
    else:
        return None

This is probably the most tempting yet dangerous mistake. Hardcoding secrets means putting sensitive information directly into your source code. While it might seem convenient because you can quickly access these secrets, it’s a security nightmare. If someone gets access to your code, they also get access to all your secrets. Plus, if you share your code with others or push it to a repository like GitHub, you might accidentally expose these secrets to the whole world. Not good!

Storing secrets in version control systems

Version control systems like Git are fantastic for managing code, but they’re not meant for storing secrets. Once a secret gets committed to a repository, it’s very hard to remove all traces of it. Even if you delete it from the code later, it still exists in the history. Attackers can easily scan repositories for sensitive credentials and steal them.

Lately, platforms like GitHub have started alerting developers when secrets are committed to version control systems which is great to see, but this doesn't stop secrets from being committed to git and leaked.

Using environment variables carelessly

Environment variables are a better option than hardcoding secrets, but they come with their own set of risks if not handled properly. Storing secrets in plaintext environment files (.env) can be risky if those files are not in your .gitignore and accidentally end up in version control. Even if the .env file doesn’t end up in version control, it’s still risky because it’s not encrypted. If an employee's laptop is accessed by an unauthorized party, the .env file and its secrets are exposed.

Additionally, .env files fail to address the secret sharing problem. Synchronizing these files across a team is challenging, leading to potential inconsistencies and security gaps. Furthermore, managing access to secrets in .env files is unfeasible, preventing effective access control.

It’s also important to ensure that your CI/CD pipelines and deployment scripts don’t inadvertently expose these environment variables. A common gotcha is leaving environment variables readable globally by any process on the system or container, rather than scoping them to just the target process.

Lack of secret rotation

Secrets, like passwords, shouldn’t be static forever. If you’re not regularly rotating your secrets, you’re increasing the risk of them being compromised. Regularly changing your secrets limits the time frame during which a stolen secret can be used. However, rotating secrets can be a hassle if not automated, so implementing a rotation policy with the help of a dedicated secret management tool is essential.

Over-permissioning

Beware of "*" permissions! Giving every application or service access to all your secrets is a bad idea. This practice, known as over-permissioning, means that if one part of your app is compromised, all your secrets are at risk. The principle of least privilege should be applied, ensuring that each part of your application only has access to the secrets it needs to function.

Inadequate auditing and monitoring

Not keeping track of who accessed your secrets and when is a big oversight. Without proper auditing and monitoring, it’s difficult to detect unauthorized access or potential breaches. Implementing logging and alerting mechanisms ensures that you can quickly identify and respond to suspicious activity involving your secrets.

This problem gets exponentially worse for larger teams. As your team grows, the number of people who need access to secrets increases, making it even more critical to have robust auditing and monitoring in place. Without these mechanisms, keeping track of access and maintaining security becomes nearly impossible, leading to a higher risk of breaches and unauthorized access.

Separation and isolation of environments

Sharing sensitive secrets across different environments is strongly discouraged because it can create several security vulnerabilities and have unintended consequences. Firstly, it may give engineers unintended access to production data. Secondly, an attacker who gains access to a vulnerable non-production environment could potentially obtain production credentials, escalating their privileges within your infrastructure. Lastly, sharing secrets and configurations across environments can complicate development, as changes in external assets such as APIs, storage buckets, or databases that your application relies on can affect all environments using the same secret.

Best practices for managing secrets

Now that we’ve covered the pitfalls and common mistakes, let’s talk about the best practices for managing your application secrets. Following these guidelines will help you keep your secrets secure and your save your engineering team countless hours grappling with hacky workarounds.

Centralize secrets

Having a single source of truth for all secrets is the first step to better secret management. This centralized source will help you reduce the number of places where secrets are stored and accessed, making it easier to manage access, deploy up-to-date secrets, employ secret rotation and monitoring.

Encrypt secrets at all times

Ensure that your secrets are encrypted at all times, and only decrypted when needed. Use strong encryption architecture to protect your secrets from being accessed or intercepted by unauthorized parties.

Control and monitor access

Use a robust access access control system to make sure only team members and machines that need access to secrets have them. Access control should take into account the unique requirements of different team members, and restrict actions like changing or deleting secrets to only those that need them. Regularly audit who has access to your secrets and monitor their usage. Implement logging to track access and changes to secrets. Set up alerts for any unusual or unauthorized access attempts. This way, you can quickly identify and respond to potential security issues.

Use versioning for secrets

When updating secrets, keep track of different versions. This allows you to roll back to a previous version if something goes wrong with a pipeline or deployment, and instantly recovery to avoid outages and downtime. It also helps in auditing changes and maintaining a history of secret usage.

Regularly rotate secrets

Secrets should not remain static. Implement a policy to regularly rotate your secrets, such as changing API keys, passwords, and certificates periodically. Regular rotation reduces the window of opportunity for an attacker to use a compromised secret.

Secure secrets across all environments

Ensure that secrets are managed and secured consistently across development, testing, and production environments. Avoid using the same secrets across these environments, and apply the same level of security to development and testing secrets as you would for production.

Limit the lifespan of secrets

Whenever possible, use short-lived secrets. Temporary secrets, like tokens or session keys, are harder to exploit because they expire quickly. This practice limits the damage that can be done if a secret is compromised.

Regularly Review and Update Secrets Management Policies

Make sure to regularly review your secrets management policies and update them as needed. Stay informed about the latest security best practices and threats, and adjust your policies accordingly to keep your secrets management up to date.

By following these best practices, you can significantly enhance the security of your application secrets and minimize the risks associated with managing sensitive information. Next, we’ll explore some tools and technologies that can help you implement these best practices effectively.

Dedicated Secret Management tools

Dedicated secret management tools such as Phase are the best way to implement all the best practices discussed above. They provide a centralized place where secrets can be stored and managed. These tools also provide the ability to enforce access control policies that restrict who has access to which secrets. They allow your team members to collaborate and work with secrets efficiently and securely without creating security vulnerabilities.

What to look for in a Secret Management tool

If you're not already using a secret management tool, you should be shopping around for one by now. Here are some key things to look for when evaluating a secret management tool.

Open source

Choose an open-source tool. Open-source software is transparent, allowing you to inspect the code and ensure there are no hidden vulnerabilities. It also benefits from community contributions and scrutiny, which can lead to quicker identification and fixing of security issues. Steering clear of vendor lock-in is also a must. This is non-negotiable.

Sophisticated encryption

Your secrets should be encrypted at all times, and only decrypted when required. Ensure the tool uses a sophisticated encryption architecture that allows both end-to-end or client-side encryption as well as server-side encryption in addition to strong, up-to-date cryptographic algorithms. A well thought-out encryption framework also facilitates granular RBAC and flexible permission models.

Developer experience

Shift security left. Putting security tools in the hands of developers is the best way to ensure secrets are handled securely from the very start of the development lifecycle. The tool should integrate seamlessly into your development workflow. Look for features like a powerful CLI, intuitive UI, SDKs, API and automation capabilities. A good developer experience increases adoption and ensures best practices are followed.

Access control and audit logging

Role-Based Access Control (RBAC) and detailed logging are critical features for any secret management tool. RBAC allows you to define who has access to specific secrets, ensuring that only authorized personnel can view or modify them. This fine-grained access control is essential for maintaining security and compliance. Additionally, comprehensive logging provides an audit trail of all secret accesses and modifications. This helps in tracking down potential security incidents and understanding usage patterns. Without robust RBAC and logging, managing secrets securely and efficiently becomes a daunting task.

Platform/Framework agnostic

Select a tool that works across different platforms and frameworks. This ensures flexibility and future-proofs your infrastructure choices. Being locked into a single ecosystem can limit your options and complicate migrations.

Integrations

A robust secret management tool should integrate well with other tools and services you use. Look for support for CI/CD pipelines, cloud providers, and various third-party services. Good integrations streamline workflows and enhance productivity.

Self-hostable

Consider a tool that offers self-hosting options. This gives you complete control over your secrets and their management infrastructure. Self-hosting can be critical for compliance with certain regulatory requirements and for organizations with strict security policies.

Extensible

The tool should be extensible, allowing you to add custom functionality or integrate with bespoke systems. An extensible tool can adapt to your evolving needs, ensuring it remains useful as your requirements grow and change. Look for platforms that offer APIs and SDKs that can be used to create integrations and solutions that meet the needs of your stack.

Conclusion

Managing application secrets is a critical aspect of modern software development. Secrets such as API keys, database credentials, and encryption keys are essential for your app’s functionality but need to be protected from unauthorized access to maintain security and reliability.

We've discussed the importance of managing these secrets properly, the common pitfalls to avoid, and best practices to follow. From hardcoding secrets to using environment variables carelessly, understanding these pitfalls helps you steer clear of potential security risks without compromising on productivity and developer experience. By implementing best practices like encrypting secrets, rotating them regularly, and using the principle of least privilege, you can enhance your app’s security posture significantly.

There are several tools available to help you manage secrets, but an open-source tool with strong encryption, a good developer-experiences and a set of critical security features should be top of your list.

In conclusion, proper secrets management is not just a best practice but a necessity in today’s develoment landscape. By adopting the right strategies and tools, you can safeguard your application’s sensitive data and ensure smooth, secure operation. Start implementing these practices today and explore Phase to take your secrets management to the next level.