SSH Key Management and Authentication

4/2/2025

This post introduces the integration of YubiKey hardware security keys for multi-factor authentication (MFA) in GitLab, leveraging FIDO2/WebAuthn protocols. It details the setup process for users to register their YubiKeys for login and explains how YubiKeys provide phishing-resistant authentication. Additionally, it covers the configuration of GPG keys on YubiKeys for signing commits, enabling verified commits in GitLab. This enhances the security of user accounts and the integrity of code contributions.

7/9/2024

This post details the RegreSSHion vulnerability (CVE-2024-6387), a remote unauthenticated code execution flaw in OpenSSH server. It explains how GitLab.com and GitLab Dedicated are not impacted due to the use of `gitlab-sshd`, which is not vulnerable. For self-managed customers, it advises applying OS patches or configuring `gitlab-sshd` for mitigation, and emphasizes general security best practices like regular patching and MFA. It also notes the low real-world success rate of the exploit.

8/17/2022

This post details the implementation of a custom SSH daemon (`gitlab-sshd`) to replace the previous OpenSSH Server and GitLab Shell setup. Key technical contributions include: enabling PROXY protocol support for Group IP address restriction, implementing graceful shutdown for Kubernetes compatibility, adding Prometheus metrics and profiling, reducing resource usage and attack surface, and simplifying the overall architecture. The post also highlights challenges encountered, such as memory consumption issues, limitations in the `golang.org/x/crypto` library (leading to a fork), and the need to re-implement OpenSSH options. The incremental rollout strategy is emphasized as a key lesson learned.

3/3/2022

Introduced support for `ecdsa-sk` and `ed25519-sk` SSH key types in GitLab 14.8, enabling users to protect their SSH keys with FIDO/U2F hardware security keys. This enhances security by requiring physical presence confirmation via a security key for SSH authentication to GitLab.

10/11/2021

This post details a vulnerability in GitKraken's key generation (versions 7.6.0 to 8.0.0) that could produce weak or duplicate SSH keys, potentially allowing unauthorized access to GitLab accounts and repositories. GitLab has responded by emailing affected users, blocking known weak keys on GitLab.com, and providing instructions for self-managed customers to revoke and regenerate their SSH keys, and for all users to update GitKraken.

5/20/2020

This post details security best practices for self-managed GitLab instances, focusing on hardening configurations. It covers strong password management for root and users, enabling Let's Encrypt for SSL/TLS, securing SSH access with support for RSA (2048-bit+) and ED25519 keys, and recommending passwordless SSH. It also outlines sign-up restrictions (disabling open sign-up, requiring email confirmation, domain whitelisting, minimum password length of 12), sign-in restrictions (requiring 2FA, disabling password authentication for Git over HTTP(S)), and visibility/privacy settings (defaulting projects to private). Additionally, it addresses network performance and stability by recommending user and IP rate limits, restricting webhook outbound requests to public services, and enabling protected paths rate limits.

10/2/2019

This post details challenges in implementing Zero Trust Networking (ZTN), specifically around user identity and device management. It highlights the difficulties in provisioning users quickly in a growing company and the unique challenge of managing devices in a BYOD-rooted culture with a standardized Linux/Mac infrastructure. The post also discusses the complexities of managing sprawling infrastructure with SSH access and the lack of two-factor authentication enforcement for SSH. Wins include the implementation of Okta for user identity management, which significantly reduced provisioning times, and the use of Chef/Knife for infrastructure changes. The team is exploring Okta's Advanced Server Access (ASA) to integrate SSH accounts and enforce multi-factor authentication, GeoIP, and other security measures for server administration.

8/27/2019

This post details debugging a scaling problem on GitLab.com related to intermittent SSH connection errors during Git pulls. The investigation involved packet captures, log analysis (HAProxy and SSHD), and identifying a pattern of errors clustering around the top of the minute, attributed to scheduled jobs hitting the `MaxStartups` limit on SSHD. The solution involved increasing the `MaxStartups` value and exploring HAProxy's rate-limiting capabilities to gracefully handle connection spikes.

5/3/2019

This post details a security incident where compromised credentials led to repository wipes and ransom demands, highlighting the importance of strong password practices and enabling two-factor authentication (2FA) and SSH keys to prevent such breaches. Mitigation strategies include using `git reflog` or `git fsck` to recover repositories and recommending secure credential management.

3/14/2017

This post details the integration of GitKraken with GitLab, enabling users to manage SSH keys, initialize new repositories, clone existing repositories, and view remote avatars directly within the GitKraken GUI. This enhances the user experience for managing GitLab repositories through a third-party client.

8/31/2016

This post details the integration of YubiKeys for two-factor authentication with GitLab, leveraging the U2F protocol to enhance security against phishing and Man-in-the-Middle attacks. It also highlights existing GitLab security features such as access and permissions control, workflow management, and audit trails, and provides nine general security best practices for Git users.

2/18/2016

Introduced an alternate `git+ssh` port (443) on GitLab.com to bypass network firewalls that block the default port 22. This was implemented by setting up a separate Azure availability set with HAProxy nodes that forward TCP connections from port 443 to port 22 of the GitLab instance, allowing users to configure their `~/.ssh/config` to use `altssh.gitlab.com` on port 443.

3/4/2014

This post introduces a screencast to demonstrate the simplicity of adding an SSH key to GitLab, aiming to alleviate user concerns about complexity and encourage its use for passwordless push/pull operations.