Hiding in your pipelines

Pwning you through your GitHub CI/CD :)

$ whoami

I’m Louis aka B611

Freelance

DevSecOps / PenTest and other stuff

Interested by this talk ?

Chat with me at louis.anelli@owasp.org 📩

GitHub Actions

tl;dr It’s the CI/CD platform of GitHub.

You write workflow files which are executed on runners 🤖

You simply put YAML files in .github/workflows/

GitHub Actions are often used to :

Build, test and deploy applications
Automate privilleged tasks
Manage settings, permissions and more in GitHub

They’re a central piece of infrastructure that can be very useful to gain access to all other resources 😋

Let’s create our first workflow

Seems simple enough !

Let’s do a workflow that displays the content of a GitHub Issue

How do we make that ?

Seems straightforward, let’s copy it

How does it work ? Let’s break it down

name: Testing triggering

on:
  issues:
  
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: View issue information
        run: |
          echo "Issue title: ${{ github.event.issue.title }}"
          echo "Issue body: ${{ github.event.issue.body }}"

Let’s manually create a test issue

The workflow works ! 🎉

Is it secure ? 🤔

I’ll ask an expert to be sure…

Content between double quotes is interpreted in bash 😱

      - name: View issue information
        run: |
          echo "Issue title: ${{ github.event.issue.title }}"
          echo "Issue body: ${{ github.event.issue.body }}"

Should be fixed 😇

      - name: View issue information
        run: |
          echo 'Issue title: ${{ github.event.issue.title }}'
          echo 'Issue body: ${{ github.event.issue.body }}'

Script injection

GitHub simply does “Match & Replace” when you use ${{ … }}

It all happens before the shell even runs :)

The code in my workflow

echo 'Issue title: ${{ github.event.issue.title }}'
echo 'Issue body: ${{ github.event.issue.body }}'

Issue to exploit the injection

After the issue creation, my code was replaced by GitHub from :

echo 'Issue title: ${{ github.event.issue.title }}'
echo 'Issue body: ${{ github.event.issue.body }}'

echo 'Issue title: Is it safe ? 🤔'
echo 'Issue body: No.';ls -la /;echo It turns out that it was not safe; printf ''

Federated OIDC credentials 👥

GitHub Secrets 🔑

Backdoors in your code/releases 💻

etc.

→ Full compromission 🤑

Supply-chain attacks

Recent RCE on openssh-server via the xz compression library made quite a bit of noise.

With never-ending dependency hell for modern frameworks and applications, it’s a good time to go hunt some dependencies :)

Supply-chain attacks

The attack surface is partically unlimited 🤑

Even if your target isn’t vulnerable, it most likely has some vulnerable dependencies somewhere on the Interwebs

Does GitHub know about it ?

yep

But they simply can’t fix it 🤯

It would be a breaking change for the millions of workflow files written on their platform.

Maintaining backward compatibility would add a lot of complexity.

Script Injection can be exploited via Issues and Pull Requests

Could anyone with read access attack you ?

The answer is… kind of. You just have to commit once before 😄

It’s not going to stop targeted attacks

It’s configurable in public repos, with a pretty relaxed default

What about workflows that trigger on GitHub Issues ? 🤔

As we saw earlier, issues can also be freely created by Readers

name: Testing triggering

on:
  issues:

No checks on issues !

Script Injection

How do we fix it ?

If GitHub themselves can’t, where do we go from there ?

Using intermediate env variables

To fix or not to fix 🤔

(inspired by true events™)

Vulnerable workflow

  - name: Show user data JSON
    run: |
      echo "${{ steps.parse.outputs.payload }}"

~~Fixed workflow~~

  - name: Show user data JSON
    env:
      user_data: ${{ steps.parse.outputs.payload }}
    run: |
      echo '${{ env.user_data }}'

Intermediate env variables can fix this vulnerability - despite adding quite a few lines - but they’re not obvious for programmers…

If creating vulns is invevitable, how can we at least detect them ? 🕵️

Semgrep SAST

Surely, someone has made a rule for this open-source code scanner ?

Semgrep rule limitations

It only scans inside a "run:", what about a "script:" ?

How about CodeQL ?

It’s the official Code Scanning tool on GitHub, they must have setup something !

CodeQL

GitHub seems to have developed something good in CodeQL, but where are the alerts ?

CodeQL scans workflows only from JavaScript repositories

No realistic solution

Semgrep	CodeQL
Incomplete rule, doesn't detect on script:	Workflow scanning only in JS repositories 😱(wontfix)
No context on user-controlled input	Still missing some things, such as indirect injections
Additional tool to configure in CI/CD	Paid on private repos

Is security a failure ?

Env variable injection

A little known way to poison pipelines 😋

Let’s create a Python application with a simple release management system and some unit tests

Just a few files and a single dependency

Some very simple tests ⚙️

import unittest
import yaml

class TestStringMethods(unittest.TestCase):

    def test_upper(self):
        self.assertEqual('foo'.upper(), 'FOO')

    def test_isupper(self):
        self.assertTrue('FOO'.isupper())
        self.assertFalse('Foo'.isupper())

    def test_split(self):
        s = 'hello world'
        self.assertEqual(s.split(), ['hello', 'world'])
        with self.assertRaises(TypeError):
            s.split(2)

    def test_yaml(self):
        self.assertEqual(yaml.dump(['foo']), '- foo\n')

if __name__ == '__main__':
    unittest.main()

Let’s setup the workflow. I want my release system to :

🚀 Start on the creation of a GitHub Issue
📖 Get the release message from the first 100 chars of the issue
⚙️ Run the unit tests
✅ Publish the release if the tests are green

How do I send the release message between steps 2 and 4 ?

Let’s set an environment variable :)

GitHub themselves recommend this method. It must be safe ! 😊

name: Automated Release Management

on:
  issues:
  
jobs:
  create_release_from_issue:
    runs-on: ubuntu-latest
    steps:
      - name: Clone the repo
        uses: actions/checkout@v4
        
      - name: Get first 100 characters of issue content for the release message
        env:
          ISSUE_NUMBER: ${{ github.event.issue.number }}
          GH_TOKEN: ${{ github.token }}
        run: |
          CLEAN_RELEASE=$(gh issue view $ISSUE_NUMBER --json body --jq .body \
          | cut -c 1-100)
          echo "RELEASE_MESSAGE=$CLEAN_RELEASE" >> $GITHUB_ENV

      - name: Run my unit tests
        run: |
          pip install -r requirements.txt
          python3 -m unittest run_tests.py -v 
      
      - name : Do the release
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run : gh release create v1.2.3 --repo "B611/script-injection-tests" \
              --notes "$RELEASE_MESSAGE"

Let’s test it out 👨‍🔬

Seems to run without errors !

Release successfully created 🥳

Can you spot the vuln ? 🔎🐛

- name: Clone the repo
  uses: actions/checkout@v4
  
- name: Get first 100 characters of issue content for the release message
  env:
    ISSUE_NUMBER: ${{ github.event.issue.number }}
    GH_TOKEN: ${{ github.token }}
  run: |
    CLEAN_RELEASE=$(gh issue view $ISSUE_NUMBER --json body --jq .body \
    | cut -c 1-100)
    echo "RELEASE_MESSAGE=$CLEAN_RELEASE" >> $GITHUB_ENV
- name: Run my unit tests
  run: |
    pip install -r requirements.txt
    python3 -m unittest run_tests.py -v 

- name : Do the release
  env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
  run : |
    gh release create v1.2.3 --repo "B611/script-injection-tests" \
    --notes "$RELEASE_MESSAGE"

Just a line break ? 😲

Not any line break…

GitHub uses by default a \r aka 0x0D in their issues, which doesn’t work.

So after some debugging I just used their API to send a \n

This was not obvious to debug as Top left is the amount of unique workflow runs :^)

The index is hitting my own server 🥳

We see the call to get the PyYAML package

And we can return anything we want

With this, we can execute arbitrary code by serving a package with a backdoored setup.py 🥰

If the packages feched are from a private artifact library (Jfrog, GitHub/GitLab, etc.), it’s even better !

We directly intercept the credentials sent within the request’s headers or body 🔑

And yet again...

Federated OIDC credentials 👥

GitHub Secrets 🔑

Backdoors in your code/releases 💻

→ Full compromission 🤑

Trying to break the GitHub CLI

Often used with high privilleges on organisations, using the identity of a GitHub App or a privilleged application.

RTFM

GH_HOST: specify the GitHub hostname for commands that would otherwise assume the “github.com” host when not in a context of an existing repository. When setting this, also set GH_ENTERPRISE_TOKEN.

This seems useful

Let’s send a malicious issue to set GH_HOST to 127.0.0.1.

If the GitHub CLI uses it correctly, the workflow should fail.

It seems to work !

Can I exfil the $GITHUB_TOKEN on a webhook ?

- name : Do the release
  env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
  run : |
    gh release create v1.2.3 --repo "B611/script-injection-tests" \
    --notes "$RELEASE_MESSAGE"

After some fighting with the format - and newlines instead of carriage returns - we manage to exfiltrate the call to a webhook !

We only have to get the GITHUB_TOKEN from the HTTP headers.

Where’s my token ???!

It’s possible to return altered data, but having the token is even better.

The release step is marked as successful despite doing nothing. The webhook was configured to answer with a HTTP 200.

After some searching, I stumbled upon this beautiful StackOverflow thread :

How to get more authenticated to github-cli?

There’s some security checks :(

const (
    github                = "github.com"
    localhost             = "github.localhost"
)

func isEnterprise(host string) bool {
	return host != github && host != localhost
}

I need to bring my own GH_ENTERPRISE_TOKEN

If the check is simply on the value host, could I simply set it to localhost and exfiltrate the token through a proxy ? 🤔

\ This can be done by setting the env variables HTTP_PROXY and HTTPS_PROXY. While they’re often used, they depend of the program’s implementation.

Seems to work in Go

Changing GH_HOST to github.localhost and github.com Changing HTTP_PROXY and HTTPS_PROXY to various values

It gives a variety of errors :

Post "http://api.github.localhost/repos/B611/script-injection-tests/releases":
http2: unsupported scheme

HTTP 404 (http://api.github.localhost/repos/B611/script-injection-tests/releases)

Post "https://api.github.localhost/repos/B611/script-injection-tests/releases":
dial tcp [::1]:443: connect: connection refused

Post "http://api.github.localhost/repos/B611/script-injection-tests/releases":
dial tcp [::1]:80: connect: connection refused

error connecting to api.github.localhosthttp
check your internet connection or https://githubstatus.com

or 🤯

error connecting to api.github.localhosthttp
lookup api.github.localhosthttp on 127.0.0.53:53: no such host
check your internet connection or https://githubstatus.com

or the classic \r created from the GitHub UI

parse "https://github.localhost\r/api/v3/repos/B611/script-injection-tests/releases":
net/url: invalid control character in URL

Too much debugging.

Isn’t there a better way in ? 🚪

GITHUB_ENV is very odd… Seems like magic 🧙 You send env vars into it, but they’re only accessible at the next step.

How does it actually work ?

An empty file with a random filename is generated at the beginning of every step.

The filepath is then set as the $GITHUB_ENV variable.

/home/runner/work/_temp/_runner_file_commands:

add_path_ad7edcc3-24e3-46b9-9c0b-b345d0d19ba6
add_path_bd446b9e-d176-4658-ad18-7de71a59c73b

save_state_ad7edcc3-24e3-46b9-9c0b-b345d0d19ba6
save_state_bd446b9e-d176-4658-ad18-7de71a59c73b

set_env_ad7edcc3-24e3-46b9-9c0b-b345d0d19ba6
set_env_bd446b9e-d176-4658-ad18-7de71a59c73b

set_output_ad7edcc3-24e3-46b9-9c0b-b345d0d19ba6
set_output_bd446b9e-d176-4658-ad18-7de71a59c73b

step_summary_ad7edcc3-24e3-46b9-9c0b-b345d0d19ba6
step_summary_bd446b9e-d176-4658-ad18-7de71a59c73b

Between your steps, the contents of this set_env_ad7e… file are read.

They are used to populate environment variables for the next step.

public string ContextName => "env";
public string FilePrefix => "set_env_";

public Type ExtensionType => typeof(IFileCommandExtension);

public void ProcessCommand(IExecutionContext context, string filePath, ContainerInfo container)
{
    var pairs = new EnvFileKeyValuePairs(context, filePath);
    foreach (var pair in pairs)
    {
        var isBlocked = false;
        foreach (var blocked in _setEnvBlockList)
        {
            if (string.Equals(blocked, pair.Key, StringComparison.OrdinalIgnoreCase))
            {
                // Log Telemetry and let user know they shouldn't do this
                ...
            }
        }
        if (isBlocked)
        {
            continue;
        }
        SetEnvironmentVariable(context, pair.Key, pair.Value);
    }
}

private static void SetEnvironmentVariable(
    IExecutionContext context,
    string name,
    string value)
{
    context.Global.EnvironmentVariables[name] = value;
    context.SetEnvContext(name, value);
    context.Debug($"{name}='{value}'");
}

private string[] _setEnvBlockList =
{
    "NODE_OPTIONS"
};

Why is there a blacklist that only contains NODE_OPTIONS ?

That’s oddly specific 🤨

This seems to be the reason, following security issues found on many repos.

NODE_OPTIONS="--experimental-modules \
--experimental-loader=data:text/javascript,console.log('123');"

Blacklisting this doesn’t solve the underlying issue.

From env variable injection to remote code execution ? 👀

Could it be possible to leverage the env variables to simply run arbitrary code on any public GitHub runner via GitHub’s code itself ?

Reminiscent of this recent Rust CVE

It most definitely is possible, to be continued… 🥰

GitHub’s security hardening page is a goldmine for attackers 🤑 Have a read

Q & A time !

Still curious about something ?

Email me at louis.anelli@owasp.org 📩