BBEdit for Log Analysis

Thursday, 19 Dec 2024

Part of the Mac series

Long-time Mac users will undoubtedly have heard of BBEdit. BBEdit is the favorite Mac text editor of many old Mac users¹ for various types of writing, including programming, notes, articles, and basically any type of writing Mac nerds can do.

Personally, I wouldn’t program using BBEdit. Visual Studio Code and offshoots like Cursor are much better for that. I wouldn’t write this blog post in it. That’s what I use Bear for. And, in my humble-ish opinion, they’re much better at those tasks than BBEdit is.

But there is a specific use that BBEdit can’t be topped for on the Mac – log file analysis. It’s like BBEdit was made specifically for it.

Here’s a UTM firewall packet filter log containing all entries for a given day, downloaded right from the UTM’s log file interface.

If I want to look at only entries for 192.168.5.5 and 192.168.5.109, I can open the Search dialog box with ⌘F and put in a regular expression to search for that.

Right in the Search dialog box, before even hitting “Find All”, it shows me at the bottom how many matches there are – 3505 of them.

If I do hit “Find All”, it opens a new window containing those lines at the top of the window, and, if I click on one of them, the log file they are in with the highlighted line in view at the bottom of the window. It’s great for contextualizing matches.

But what if I want a document that contains those matching lines and nothing else? That’s what the “Extract” button on the Search dialog box is for. By clicking that instead of “Find All”, a new document is created in BBEdit that contains just the matching lines named “Extracted occurrences of (insert regular expression search string here)”.

I use this all the time to drill down further and further without having to edit any files or keep search results windows open. Let’s say I have a subset of IP addresses extracted, like anything in the 192.168.5.x range, and I want to see only lines that have a firewall action of “drop” in them. In other words, I want to ignore all cases where the firewall passed the traffic along and only see when it blocked it for those IP addresses.

Very simple. While viewing the document of extracted 192.168.5 matches, ⌘F again and enter a new regular expression looking for lines where action=“drop”. The search dialog shows 29 matches.

Hit the “Extract” button and extract the 29 matches to a new document.

Sometimes when viewing results like this I want blank lines between them so I use a regular expression find and replace when doing this, and click the “Replace All” button in the search dialog, which acts on the active document and replaces all matches with what I want – in my case, the same result lines but followed by 2 newlines after each of them.

Much better for my tired old eyes.

And now for some additional niceties of BBEdit search:

1. You can save your searches and access them using the “g” button on the Search dialog to get a dropdown list to choose from.

2. You can also, as my examples in this post illustrate, pull up previously used searches from the search history feature by clicking the clock icon button on the search dialog, and choosing from the dropdown list.

3. There’s also a multi-file search you can open separately from regular search, with the same features mentioned above, using ⇧⌘F. It lets you choose the files to search in with great granularity, and it has the same “Find All”, “Replace All”, and “Extract” functions as the regular active file search function. This means you can extract all instances of a specific thing contained in multiple files to one new document.

One important thing about composing a search in BBEdit if you do want to extract to a new document is that you need to compose your regular expression to include a full line and not just the exact search terms specified. For example, if you just search for “192.168.5” like this, and then Extract:

You get this:

Not very useful if you’re into context or understanding of any kind. But if you use a regular expression that looks like this (since in my case, the log is separate lines of logged events):

Now you get the full line for each matching line:

That’s because the search was created with a regular expression of ^.*192\.168\.5.*$, which looks for start of line, any characters any number of times followed by 192.168.5, followed by any number of any characters, followed by end of line. When using regular expressions as search terms, make sure that the Grep option next to “Matching:” in the search dialog box is checked so that the search term acts as a regular expression instead of a literal string.

I’m sure there’s much more that I haven’t even discovered about BBEdit and searching and extracting yet, but these are some things that I use all the time and find indispensable. If your job or hobby requires you to parse log files very often, I recommend a copy of BBEdit even if you never use it for anything else. It’s still very worth the price for that.

Part of the Astro series

Astro 5 is out, and it has a number of changes from Astro 4. The good news is, I was able to upgrade this site to it without making any changes at all. That doesn’t mean I don’t have any to make – I need to convert my Content Collections to the new Content Layer API at the very least, for example.

But the fact that it worked as is enabled me to update all my packages to the latest versions and then worry about updating specific implementation details later.

I do want to point out that if you have any sites using the Astro Starlight documentation site framework, you can’t upgrade to Astro 5 yet. It will not compile. There is a PR in place and the stellar Astro documentation team is all over it!

I’ll make notes about the actual changes I do make to this site to be fully Astro 5 compliant and write a post about it soon.

Contents

• GitHub Deploy Keys
• Create the Deploy Keys
• Add the Public Keys to the GitHub Repositories
• Assign the Correct Keys to the Correct Repositories

GitHub Deploy Keys

Sometimes I want to be able to pull a personal GitHub repository onto a server in order to keep some files up to date. Things like scripts that I won’t edit on the server and so won’t need to push to GitHub are good candidates for this. In order to accomplish this in a reasonably secure way, I use GitHub Deploy Keys. With a read-only deploy key, if someone ever gets access to my server I’ll suddenly have a billion new problems, but worrying about my repositories being overwritten with malicious garbage won’t be one of them.

Deploy keys are repository specific, however, which means if you have multiple repositories you want to pull from GitHub onto the server, you need a way to specify which deploy key each repo should use.

Let’s use an example where I have two repos, both of which I want to pull onto my server. One is called server-management-scripts and the other is called website-management-scripts.

Create the Deploy Keys

You can use good old ssh-keygen in your account’s .ssh directory on the server to create your keys. In my case, ecdsa keys, named sms-deploy (for server-management-scripts repo) and wms-deploy (for website-management-scripts repo).

ssh-keygen -t ecdsa -f sms-deploy

And

ssh-keygen -t ecdsa -f wms-deploy

Because these are going to be read-only keys for their repos, I do not use a passphrase on them.

scott@dragonfly ~/.ssh $ ll
total 40
drwx------ 2 scott scott 4096 Nov 19 18:23 ./
drwxr-xr-x 8 scott scott 4096 Nov 19 18:21 ../
-rw------- 1 scott scott  194 May  8  2024 authorized_keys
-rw-r--r-- 1 scott scott  133 Mar 12  2021 config
-rw------- 1 scott scott  505 Nov 19 18:22 sms-deploy
-rw-r--r-- 1 scott scott  177 Nov 19 18:22 sms-deploy.pub
-rw------- 1 scott scott  505 Nov 19 18:23 wms-deploy
-rw-r--r-- 1 scott scott  177 Nov 19 18:23 wms-deploy.pub

Add the Public Keys to the GitHub Repositories

Go to your repository on GitHub, go into the repository settings, Click the Deploy Keys link, and click the Add Deploy Key button.

Give the key a name in the Title textbox, copy the contents of your public key (the .pub file for the key) that you created on your server into the Key textbox, and leave “Allow write access” unchecked. Click the “Add key” button.

Now your new deploy key shows in the repo’s Settings > Deploy keys, and this key can be used to pull the repo.

Once you’ve added the public keys to the repos, you can delete them from your server, leaving only the private keys behind.

scott@dragonfly ~/.ssh $ ll
total 32
drwx------ 2 scott scott 4096 Nov 19 18:23 ./
drwxr-xr-x 8 scott scott 4096 Nov 19 18:21 ../
-rw------- 1 scott scott  194 May  8  2024 authorized_keys
-rw-r--r-- 1 scott scott  133 Mar 12  2021 config
-rw------- 1 scott scott  505 Nov 19 18:22 sms-deploy
-rw------- 1 scott scott  505 Nov 19 18:23 wms-deploy

Assign the Correct Keys to the Correct Repositories

In order to make sure that a git pull command works correctly in each repo, I can edit my user account ssh config file on the server to contain these entries:

Host serverscripts
 HostName github.com
 User git
 AddKeysToAgent yes
 IdentityFile ~/.ssh/sms-deploy
 IdentitiesOnly yes
 ForwardAgent yes

Host websitescripts
 HostName github.com
 User git
 AddKeysToAgent yes
 IdentityFile ~/.ssh/wms-deploy
 IdentitiesOnly yes
 ForwardAgent yes

Now, in each repo’s .git directory on the server, I can edit the config file to reference the correct entry in ~/.ssh/config :

[core]
 repositoryformatversion = 0
 filemode = true
 bare = false
 logallrefupdates = true
[remote "origin"]
 url = git@serverscripts:scottaw66/server-management-scripts.git
 fetch = +refs/heads/*:refs/remotes/origin/*
[branch "main"]
 remote = origin
 merge = refs/heads/main

Notice the highlighted part, which is the URL for the repo to use. It uses serverscripts as the server portion of the URL (the part after the @ symbol and before the :), because that matches the first Host entry in my account’s ~/.ssh/config file on the server.

The second git repo’s config file on the server would look like this, since I gave it a Host name of websitescripts in the ~/.ssh/config file:

[core]
 repositoryformatversion = 0
 filemode = true
 bare = false
 logallrefupdates = true
[remote "origin"]
 url = git@websitescripts:scottaw66/website-management-scripts.git
 fetch = +refs/heads/*:refs/remotes/origin/*
[branch "main"]
 remote = origin
 merge = refs/heads/main

Now both repos are linked back to a specific entry in ~/.ssh/config, and therefore to their respective deploy keys, specified on the IdentityFile line. The ssh key file for the server management scripts repo is ~/.ssh/sms-deploy, and the ssh key for the website management scripts repo is ~/.ssh/wms-deploy.

When you do a git pull in either of those repos on the server, it will now use the correct deploy key to authenticate.

Initial SuperDuper Copy

Monday, 28 Oct 2024

Part of the Mac series

Last night I mentioned setting up SuperDuper! to clone my Mac’s internal SSD to an external SSD daily. One thing about the initial copy you should know that doesn’t pertain to subsequent incremental copies is don’t let your Mac sleep during the initial copy.

I even thought about this when I started the initial backup, but then thought it shouldn’t matter. I came back to a failed copy with the following errors in the SuperDuper! log:

| Info |       Volume replication failed - Resource busy
| Error | ***ERROR OCCURRED: ****FAILED****: result=256 errno=60 (Operation not permitted)

I sent the log to Shirt Pocket support, and Dave Nanian quickly replied with the following (I hope he doesn’t mind me posting it here):

Your Mac has fallen asleep during the copy, even though we asked it not to, or something similar (such as a TRIM operation that takes too long) — as you can see, Apple’s error is vague and unhelpful.

But, in our experience, this usually works. Please reformat the backup drive (which, done this way, should reset all the TRIM operations):

• Open Disk Utility
• Choose “Show All Devices” from the View menu
• Select the destination drive hardware (above the existing volume)
• Click Erase
• Choose the “GUID” partition scheme (2nd pop-up), THEN “APFS” formatting (1st pop-up) and name appropriately
• Click Erase

Then restart your Mac. Install Coca (free) from the App Store and enable it to keep your Mac awake — including the screen. Ensure it’s plugged in if it’s a laptop. Turn Time Machine off temporarily. Then try the erase-then-copy backup again.

Note that none of this should be necessary for future Smart Updates, which are done with our copy engine, and not Apple’s.

Hope that helps!
—
Dave Nanian
Shirt Pocket

In my case, I use the Raycast Coffee extension now when I want to keep my Mac awake, so I used that to make sure the first copy finished.

Anyway, don’t sleep on that first copy, and do back your drive up to more than one destination. Make at least one local and at least one remote (cloud backup is fine). I highly recommend SuperDuper! for its features, price, and always outstanding support from Dave Nanian.

SuperDuper Invisible

Sunday, 27 Oct 2024

Part of the Mac series

I’ve certainly used drive cloning software in the past, including SuperDuper!.¹ For awhile now, though, I’ve just had Backblaze as my backup method, which does violate my principle of always having local and remote backups. So when I saw Jason Snell’s article on SixColors about using a cloned drive to recover from Mac failures, it gave me the impetus to make a clone again.

I love and like and love SuperDuper! and always have, but I do wonder about this UI choice…

If you can’t see the blue progress bar inside the outlined section of that Copy Files section header, you are not the only one. It took me a couple minutes to realize it was there.

Look, I may be the pot calling the kettle black, or not picking the mote out of my own eye, or whatever metaphor from 50,000 years ago for hypocrisy you want to use because of the color of the hyperlinks on my site’s dark theme, but that is some kind of invisible ink level trickery on that progress bar.

Anyway… according to the invisible progress bar, as I write this SuperDuper! is about 75% of the way done cloning my internal drive.

You all know, I hope, that I have a /now page on this site for quick dumps of things I’m into at the moment. It’s ephemeral, as /now pages should be.

The idea of the /now page comes from Derek Sivers, and he has a whole site, nownownow.com, dedicated to spreading the message and the sheer fun of /now pages.

And look! You can see people by region!

I’m very pleased to announce that, as of this writing, 37 of us in Oregon are extremely proud of whatever it is we’re doing /now. You should definitely check it out. I’m even listed right beside Justin Miller.

I was a little surprised to see that everyone in Japan touting their /now pages are gaijin¹ and not Japanese. Some of those people in Japan need to talk to the people they live amongst and spread the news! Richard Möhn is my hero though – he lives in Kagoshima City, my old stomping grounds!

Here’s your assignment for today:

• Visit my /now page.
• Read up about /now pages.
• Make your own.
• Email the URL to Derek Sivers.
• Be part of a nice group of people who like to have fun, love the indie web, and think you should do it /now!

Mail Weekly Reads Script Command

Thursday, 17 Oct 2024

Part of the Raycast series

As I mentioned previously, I have a Weekly Reads page that I set up to make it easier for me to share articles with my dad and brother (and anyone else who cares), complete with RSS feed. But let’s be real, neither of them wants to be bothered with my feed because they have stuff to do and we already have a tradition of emailing links to each other.¹

Last night I created a Raycast Script Command that looks at the markdown file for my latest Reads update, parses and reformats it for email, and mails it out to them. It looks like this:

#!/bin/bash

# Required parameters:
# @raycast.schemaVersion 1
# @raycast.title Mail Weekly Reads
# @raycast.mode fullOutput

# Optional parameters:
# @raycast.icon
# @raycast.argument1 { "type": "text", "placeholder": "Subject" }
# @raycast.packageName Website

# Documentation:
# @raycast.author scott_willsey
# @raycast.authorURL https://raycast.com/scott_willsey

subject=$1

# Find the latest file in the specified directory
latest_file=$(ls -t ~/Sites/scottwillsey/src/content/reads | head -n1)
echo "Processing file: $latest_file"

# Read the contents of the file, remove YAML front matter, and modify markdown links
content=$(awk '
    BEGIN { in_frontmatter=0; }
    /^---$/ { in_frontmatter = !in_frontmatter; next; }
    !in_frontmatter { print; }
' "/Users/scott/Sites/scottwillsey/src/content/reads/$latest_file")

content=$(echo "$content" | perl -0pe 's/\*\*\[(.+)\]\((.+)\)\*\*\n\n(.+)/$3 → $1 – $2\n/gm')

echo "Content after regex replacement:"
echo "$content"

# Escape the content for AppleScript
escaped_content=$(echo "$content" | sed -e 's/\\/\\\\/g' -e 's/"/\\"/g' -e 's/`/\\`/g' -e "s/'/\\'/g" -e 's/$/\\n/' | tr -d '\n')

# Pass the escaped content to the AppleScript
osascript <<EOD
on run
    set theContent to "$escaped_content"
    tell application "Mail"
        set theFrom to "me@mine.com"
        set theTos to {"dad@dads.com", "brother@brothers.com"}
        set theCcs to {}
        set theBccs to {}

        set theSubject to "$subject"
        set theSignature to ""
        set theAttachments to {}
        set theDelay to 1

        set theMessage to make new outgoing message with properties {sender:theFrom, subject:theSubject, content:theContent, visible:false}
        tell theMessage
            repeat with theTo in theTos
                make new recipient at end of to recipients with properties {address:theTo}
            end repeat
        end tell

        send theMessage
    end tell
end run
EOD

It takes one parameter when I run the script so that I can give the email a cutesy custom subject (another idea I stole from my brother).

Then it parses the latest file in my web project’s /src/content/reads directory, replaces markdown links with “commentary → link title – link” for each link and following commentary paragraph in the page, and gives that to an AppleScript to send the mail.

From this markdown page that generates the individual HTML reads page:

To this email that gets sent to my long-suffering family members:

It’s simple and effective. You can do this and make your family hate you too.