Last week I said it was time to figure out what kinds of automated tasks it actually made sense to have an AI agent do for you. Thinking about this kind of made my path within the newsletter bifurcate and I think splitting my attention up makes sense, but we’ll see.
On the one hand, I want to continue exploring and developing the mechanisms used to route local prompts to the best model for the job (including cloud models anytime that’s needed). And that’ll be going on full steam, even though this particular newsletter issue won’t have that much to say about that.
At the same time, I think the expedient way to try out and build various kinds of tasks and skills is to do it on a fully backed system, namely Claude Cowork. Working concepts and flows can be shifted to local tools as appropriate…later. So this week we’ll tackle how that’s gone so far.
A Real History
One thing I’d really like an “assistant” to do for me is be aware of my history across most or all aspects of my life over time. That’s a lot of information over time, and we can’t just wave our hands in the air and say “context is king” because this is a lot more info than we’d ordinarily want to just throw up into context for the current prompt. Indeed, this is a lot more context than the current models can hold. So we’ve got to be thinking about how to manage loading relevant slices as needed.
This assumes we’ve got the info in the first place, and this is where I wanted to start my experiment. There are plenty of things in my life where the online work I’m doing creates a record of some kind, either literally as system logs or as accumulated documents, web pages, and github repositories. But plenty of other things I feel like I just need to “tell” the assistant.
The Talking Cure
So toward that end, I thought: why not just talk to it and give it info and updates in short snippets all day long. Get in the habit of making a voice note of everything. I could, I figured, easily use my iPhone for this. I’d give it a line or two and it could store that in a file in the cloud. I could then make a recurring task in Cowork that read and synthesized each day’s file. Each morning it could send me a rundown of relevant notes, reminders of dates I read into the notes, and presumably it could also carry out simple tasks straight from the notes “Please find out whether the Call for Papers deadline at X has been extended and note that in the report”.
This is just a beginning point, I should add. My gut tells me I’ll need to come up with some kind of process where AI overlays some kind of classification scheme. But the point is to start with something that kind of does something useful, then experiment, learn, and iterate from there.
To get going, I realized that it obviously had to be easy easy easy to input new notes. I asked Claude for suggestions and the suggestion I got was to use the Shortcuts feature of iOS to create a shortcut that would, when run, just throw up a voice input prompt that rendered the input as text, stopped at the first pause, and posted it as a new entry in a text file named with today’s date. It gave me instructions for how to do this.
Shortcuts: Sucktastic but Useful
I hadn’t bothered with Shortcuts in the past and quickly learned that it’s living in its own quirky macro language universe. Having spent a lot of time with quirky macro systems back in my first years as a professional developer, I started to see the logic of how it worked, but let’s just say that it only came together with a lot of back and forth with Claude and me repeatedly pointing out that whatever it had just suggested didn’t actually work.
To execute the shortcut, I first fiddled around with assigning the shortcut to the “Back Double Tap” input of the iPhone (yes, you can double or triple tap the back of your phone and it kicks off whatever you’ve assigned it to… who knew?). The cover I have on the phone makes it pretty difficult for the taps to be picked up by the phone (the point of the cover is to shield the phone from unwanted “taps,” after all).
Double Tapped
I then assigned the shortcut to the lock screen of the phone, so I can just tap that and the prompt comes up instantaneously. Perfect. And I put the icon for the shortcut on the home screen so it was right there if the phone was already unlocked.
I’m running on an aging iPhone 14, by the way. If I had a 15 or later, I could also assign the shortcut to the physical “action button” and it would just launch, whether the phone was awake or not. Might be time to take one of those free upgrade deals that lock you into a contract till the end of time.
I’ve got the recipe for the shortcut posted as a Field Note on PeakZebra.com, so you can chase this down yourself if you’re inclined. I know next to nothing about Android, but I assume there’s a way to pull off the same trick.
The file winds up in an iCloud folder and I gave Claude access to this folder so it could read the text files. Here’s the skill.md file it uses for the recurring morning task:
morning-voice-note-digest prompt:
You are Robert's personal assistant. Your job is to read his voice notes from the past 24 hours and produce a concise, useful morning digest.
Notes location: Robert's voice notes are stored as plain text files in:
C:UsersRuggeiCloudDriveVoice NotesFiles are named YYYY-MM-DD.txt.txt (double extension) and contain timestamped entries in the format:
M/D/YY, H:MM AM/PM: [note text]
What to do:
1. Read yesterday's note file (and today's if it already exists). If no file exists for yesterday, note that and continue.
2. Extract and categorize: commitments/follow-ups, decisions made, open questions, people who need a response.
3. Advise on task timing — flag anything time-sensitive or with an implied deadline.
4. Search for relevant news on 2-3 key topics from the notes. Only surface what would genuinely affect Robert's decisions.
5. Output sections: Today's priorities (top 3) / Don't forget / Decisions logged / Relevant news (only if useful)
Keep it tight — 2 minutes to read. Be direct and specific.Here’s an early version of the digest, where I’m very interested to see that it read back fairly far into the files, even though it wasn’t expressly asked to, and caught that I’d missed a deadline I’d mentioned a week before:
Morning Digest — Monday, June 1, 2026
Today's priorities
1. WordCamp US call for papers — noted May 25 that the deadline was May 29. That's passed — worth checking if you submitted or if there's still a late window.
2. Fix the dangerous screw — noted May 25, no follow-up logged. Physical hazard, easy to forget.
3. Look into LTX Studio (ltx.studio) — flagged as interesting on May 25, not revisited.
Don't forget
* Check on the micro video production post you found interesting (May 25, no link saved)
* Poison ivy noted today at 2:13pm — keep an eye on it
Decisions logged
* Connected iCloud Drive sync to local notebook
* Granted Cowork access to Voice Notes folder ✓
Relevant news — Skipped (today's notes are setup/admin; nothing to search against yet)The thing that jumps out at me is that it is patently ridiculous that you’d have to remind anyone to pay attention to their poison ivy rash. Trust me, it sends its own reminders.
There’s nothing of interest noted after May 25 because of some problems and bugs I had to sort out so that the files wound up in the right place and the daily task was looking in the right folder for them. And by now I have no idea what “Fix the dangerous screw” means, but I was doing some work on a small farm I help run (story for another day and another newsletter, I guess), and it probably really was dangerous. Hopefully I did in fact fix it.
The imperfect part of this is that I have to remember to talk to the iPhone regularly. This may take some cultivation. Otherwise, what I’m imagining is that I begin adding access to records that indicate what I’m up to — emails probably first, then, I dunno, bank account records?
I’m sort of kidding about bank account records being added so soon, but I do think ultimately that’s the end game. In the interim, though, maybe the next thing after email will be the to-do list I keep on the Todoist app.
But Wait! It Doesn’t Work!
When I checked my morning digest the morning after I finished setting things up, it had failed to find the text files in the Voice Notes directory I’d created. They were there, of course — I’d checked.
A little back and forth with Claude Cowork established that Cowork can only see the folder that’s currently in context within the current session. It’s kind of the same concept as the current working directory, if you’re familiar with this from command-line driven operating system environments. If you are working in the directory Cowork established with you as it’s primary directory, it won’t see any other directory. And you can’t give it instructions in your scheduled task to change directories. Call me crazy, but I think this is a flaw.
This won’t be a problem if I implement something along the same lines on OpenClaw, which has access to the full local filesystem, but for my money it means you can’t do anything useful in local directories using a scheduled Cowork task.
Enter Airtable
The solution has proven to be writing my notes to a table in my Airtable account. I can give Cowork access to any “bases” and tables that I want in Airtable, so this just works. It is not in the least bit painless to set up Shortcuts to write to the table, so if you’re contemplating doing this, I heartily recommend checking out my Field Notes on this at PeakZebra.com.
Clawing My Way Back
While working on that, I’ve been circling back to OpenClaw, something I’d set aside because I wanted to work with Ollama and local models more directly. But I’m swinging back around because some interesting routing tasks and approaches have been created and I want to find out whether I still need to write my own router. What’s different about these routers is that they tend to use the cloud model to decide the routing path, whereas I wanted to do much or all of the routing using simpler machine learning and, where needed, a local model. More to come on all this.