[GHAS CodeQL Series] - Part 3: Blocking Vulnerable Code Merges

[vishaljsoni]

Welcome to the final installment of our GitHub Advanced Security CodeQL series! In Part 1 you established organization-wide scanning, and in Part 2 you implemented alert-mode rulesets to build security awareness. Now it's time to complete your security implementation by enabling blocking controls that prevent vulnerable code from entering your codebase.

This final step implements the "stop the leaking bucket" philosophy: while your teams work to remediate existing vulnerabilities, we'll ensure no new ones are introduced.

Series Overview

• Part 1: Setting Up Organization-Wide Code Scanning ✅
• Part 2: Implementing Alert-Mode Repository Rulesets ✅
• Part 3: Blocking Vulnerable Code Merges (This post)

The "Stop the Leaking Bucket" Philosophy

Think of your codebase as a bucket with holes representing security vulnerabilities. While it's important to patch existing holes (fix current vulnerabilities), it's even more critical to prevent new holes from appearing (block new vulnerabilities).

Why Blocking Controls Matter:

• Prevents Security Debt: Stops accumulation of new vulnerabilities
• Creates Security Culture: Developers learn secure coding in real-time
• Provides Immediate Protection: Vulnerable code never reaches production
• Maintains Baseline: Preserves security posture improvements over time

Upgrading Your Alert-Mode Ruleset to Blocking Mode

Since you already have an alert-mode ruleset from Part 2, we'll simply modify it to enable blocking behavior.

Step 1: Access Your Existing Ruleset

1. Navigate to your organization's Repository rules page
2. Find your existing "CodeQL Alert Mode" ruleset
3. Click Edit to modify the configuration

Step 2: Configure Blocking Thresholds

This is where we transform alerts into blocking controls. Under the "Require code scanning results" section for CodeQL:

Alert Severity (General Findings)

• Current Setting: "None" (alert mode)
• Recommended Update: "Errors" (blocks error-level findings)
• Alternative Options: "Errors and Warnings" or "All" (more restrictive)

Security Alert Severity

• Current Setting: "None" (alert mode)
• Recommended Update: "High or higher" (blocks high/critical security issues)
• Alternative Options: "Medium or higher" or "All" (more comprehensive)

Step 3: Save and Deploy

1. Click Update to save your changes
2. The updated ruleset applies immediately to all targeted repositories
3. Monitor the first few hours for any unexpected blocking behavior

Testing Your Blocking Implementation

Verify your blocking controls work correctly with controlled testing.

Create Test Vulnerabilities

Use these examples to test blocking behavior:

JavaScript SQL Injection Test

// test-vulnerability.js
const mysql = require('mysql');

function getUserById(userId) {
    const connection = mysql.createConnection({
        host: 'localhost', user: 'app', password: 'password', database: 'users'
    });
    
    // Vulnerable: Direct string concatenation
    const query = "SELECT * FROM users WHERE id = " + userId;
    connection.query(query, (error, results) => {
        console.log(results);
    });
}

Python Command Injection Test

# test-vulnerability.py
import subprocess
import sys

def process_command(user_input):
    # Vulnerable: Direct execution of user input
    result = subprocess.run(f"echo {user_input}", shell=True, capture_output=True)
    return result.stdout.decode()

Verification Process

1. Create test branch with vulnerable code

2. Open pull request to protected branch

3. Verify blocking behavior:

   • CodeQL scan detects vulnerability
   • Code scanning merge protection blocks the PR from merging
   • Merge button is disabled
   • Security alerts appear as annotations in the PR

4. Test resolution workflow:

   • Fix the vulnerability (or use Copilot Autofix to generate a suggested fix)
   • Push changes to update PR
   • Verify scan passes and merge becomes available

Managing Exceptions and False Positives

Dismissing False Positives

When encountering legitimate false positives, you have two options:

Option 1: Dismiss directly from pull request annotations (recommended)

When code scanning detects issues in a PR, alerts appear as annotations in the Files changed tab. You can dismiss an alert directly from the annotation by clicking Dismiss alert and selecting the appropriate reason. This is the quickest approach when reviewing PR findings.

Option 2: Dismiss from the Security tab

1. Navigate to Security tab → Code scanning alerts
2. Click the specific alert → Dismiss alert
3. Select appropriate reason:
   • False positive: Incorrect identification
   • Won't fix: Valid finding but accepted risk
   • Used in tests: Intentional test vulnerabilities
4. Add detailed justification comment

Note: Organizations can enable delegated alert dismissal to require a reviewer to approve or reject dismissal requests, ensuring stronger governance over dismissed findings.

Bypassing Merge Protection

For situations where a PR needs to merge despite failing code scanning checks, repository rulesets provide a Bypass List that controls who can bypass the ruleset enforcement.

Configuring the Bypass List

1. Navigate to your organization or repository ruleset settings
2. Under the Bypass list section, add the roles, teams, or GitHub Apps that should be allowed to bypass the ruleset
3. When a user on the bypass list merges a PR that would normally be blocked, they are prompted to provide a reason for the bypass
4. All bypass actions are logged in the audit log for traceability

Best Practices for Bypass Management

• Keep the bypass list small: Limit to security leads, team leads, or a dedicated security review team
• Require bypass justification: All bypass merges should include a documented reason and a remediation timeline
• Monitor bypass usage: Regularly review audit logs for bypass activity to identify patterns that may indicate rules need adjustment
• Use temporary bypasses sparingly: For critical production incidents, add a team temporarily and remove after the incident is resolved

Known Limitations

Be aware of these limitations when relying on code scanning merge protection via rulesets:

• Merge queue groups: Merge protection rulesets do not apply to pull requests merged through merge queue groups
• Dependabot default setup: Dependabot pull requests analyzed by default setup are exempt from code scanning merge protection
• PR diff scope: For a code scanning alert to block a PR, all lines identified in the alert must exist in the pull request diff

For the most current information on limitations, see Code scanning merge protection.

Monitoring and Success Metrics

Track these key indicators after enabling blocking:

Security Metrics

• Blocked merges per week: Volume of prevented vulnerabilities
• Time to vulnerability resolution: Speed of security issue fixes
• Alert accuracy rate: Valid findings vs. false positives
• Security debt trend: Overall vulnerability count trajectory

Developer Experience

• Average merge delay: Impact on development velocity
• Developer satisfaction: Feedback on security workflow integration
• Exception request frequency: Indicator of appropriate rule configuration

Recommended Monitoring Schedule

• Daily: Monitor blocked PRs and developer support requests
• Weekly: Review metrics trends and false positive patterns
• Monthly: Assess threshold appropriateness and adjust as needed
• Quarterly: Evaluate overall security posture improvements

Troubleshooting Common Issues

High False Positive Rates

Solution: Review and adjust severity thresholds, provide developer training on dismissal procedures

Developer Resistance

Solution: Emphasize security benefits, provide clear remediation guidance, establish responsive support channels

Frequent Bypasses

Solution: Review bypass list membership, strengthen approval processes, address underlying workflow issues, improve developer security training

Phased Rollout for Large Organizations

For organizations with many repositories, consider gradual deployment:

1. Week 1-2: High-maturity security teams (10-20% of repositories)
2. Week 3-4: Critical production services and main applications
3. Week 5-6: Organization-wide deployment with comprehensive support

Tip: Use the ruleset Evaluate mode before switching to Active enforcement. In Evaluate mode, the ruleset tracks which merges would have been blocked without actually preventing them. This lets you fine-tune severity thresholds and identify false positive patterns before they impact developer workflows. Review the results in the Ruleset Insights dashboard, then switch to Active when confident. This approach is recommended by the GitHub Well-Architected Framework.

Congratulations! Your Security Program is Complete

You've successfully implemented a comprehensive GitHub Advanced Security program with CodeQL! Here's what you've accomplished:

Your Journey Summary

• ✅ Part 1: Established organization-wide CodeQL scanning foundation
• ✅ Part 2: Built developer security awareness through alert-mode rulesets
• ✅ Part 3: Implemented blocking controls to prevent new vulnerabilities

The Security Culture You've Created

• Proactive Security: Vulnerabilities caught before production
• Developer Education: Teams understand security implications of code choices
• Quality Gates: Security integrated into development process
• Risk Reduction: Continuously shrinking organizational attack surface

Advanced Next Steps

Your foundational program is complete. Consider these enhancements:

• Copilot Autofix for AI-generated fix suggestions on code scanning alerts
• Custom CodeQL queries for organization-specific security rules
• Delegated alert dismissal for governance over alert dismissals across your organization
• Integration with SIEM systems for broader security monitoring
• Automated remediation workflows for common vulnerability patterns
• Security champion programs for continued developer education

Key Takeaways

• Progressive implementation (alert → blocking) maximizes success rates
• Repository rulesets provide scalable, modern governance superior to legacy approaches
• Proper testing and exception handling ensure developer productivity alongside security
• Continuous monitoring and threshold adjustment are essential for long-term success

Essential Resources

• GitHub Docs: Code scanning merge protection (concepts)
• GitHub Docs: Set code scanning merge protection
• GitHub Docs: Repository rulesets
• GitHub Docs: Triaging code scanning alerts in pull requests
• GitHub Well-Architected: Rulesets best practices

---

Congratulations on completing our GitHub Advanced Security CodeQL series! You've built a comprehensive, scalable security program that protects your organization while empowering developers. Your progressive implementation approach sets the foundation for long-term security success.

Series Navigation:

• Part 1: Setting Up Organization-Wide Code Scanning
• Part 2: Alert-Mode Repository Rulesets
• Part 3: Blocking Vulnerable Code Merges (You are here)

🔒 Thank you for making the software world more secure! 🔒

[Bhanu99517]

Excellent wrap-up of the series—this effectively captures the shift from reactive to preventative security by operationalizing CodeQL as an enforceable quality gate rather than just a visibility tool.

What stands out is the emphasis on progressive enforcement (alert → evaluate → blocking), which aligns well with modern DevSecOps maturity models. The “stop the leaking bucket” analogy is simple but accurately reflects the need to control vulnerability ingress while remediation efforts are ongoing.

From an implementation perspective, the guidance around severity threshold tuning and bypass governance is critical. In practice, the success of blocking controls heavily depends on maintaining a balance between signal quality and developer velocity. High false-positive rates or overly aggressive thresholds can quickly erode trust in the system, leading to excessive bypass usage—so the recommendation to monitor audit logs and iterate on thresholds is spot on.

One area that could be further explored is integration with developer workflows at scale, such as:

• Automating remediation using tools like Copilot Autofix or custom fix pipelines
• Establishing SLAs for vulnerability resolution tied to severity levels
• Leveraging metrics (e.g., MTTR for security findings, bypass frequency trends) to continuously refine policy enforcement

Additionally, highlighting how these rulesets interact with edge cases like merge queues and Dependabot exemptions is valuable, as these nuances can impact real-world enforcement guarantees.

Overall, this provides a strong, production-ready foundation for organizations aiming to embed security directly into their CI/CD pipelines while maintaining governance and scalability.

[vishaljsoni]

Agred, working towards the metrics and expanding to Security Champions program is a natural next steps from this.