When I was thinking of training a personal AI Agent, I wanted to make sure it can be actually useful. I went over about 100 of my real actual tasks for the month, analyzed which parts an AI can do and generalized into 12 categories. Here are the categories, what I successfully implemented and what was the biggest impact in the four months I’m using it. The “requirements” part is brought almost perfectly copy pasted, including question marks about some of them and other hesitation or thoughts buried in them.
Requirements:
- Each conversation should be approved in a Telegram channel
- Each plan conversation should be approved including some back and forth plan in Telegram
What I implemented:
- “Air-gapped” read of all WhatsApp <link to previous blog>
- Didn’t implement write for now as I feel that AI writing as me will be too socially unacceptable for now. Some apps I wrote have an intent that opens WhatsApp with a message.
- I gave him his own WhatsApp and reimplemented the WhatsApp handling of OpenClaw with tougher controls – It must confirm the exact message with me on Telegram before sending messages to anyone except itself and me. It can’t disclose my information or even be a helpful assistant. More on this in an upcoming blog.
Real world impact: Big
- Reminders of items that I didn’t create calendar reminders or todos for – saved me a lot of time, for example to bring items to kids’ kindergartens, doctor appointments that spouse set up but didn’t send an invite, etc.
- Summaries of conversation for later review – more thoughtful relationships and shortening the time I forget to answer friends where a conversation tapered off.
- Follows WhatsApp groups that I don’t want to follow and can’t get out of like kindergarten groups that only rarely have important information.
- It contacts shops and service providers for me in WhatsApp to ask questions or make sure things are in stock. This is amazing.
- It can create summaries of things that my friends asked me and send it to them if I allow it, for example a friend asked how to install OpenClaw and configure it and I just told it to send it to her without needing to copy paste things around. For now friends and family that interacted with it are very positive and semi-interested only. Maybe it says something about me that this is not super surprising to people that I got a bot to message them 🙂
Future improvements:
- Currently the WhatsApp integration is a Tampermonkey userscript that needs me to run WhatsApp Web. This is both fragile and not accurate if I step away from the computer for vacation or whatever. Need to improve this to be truly independent.
Personal Info
Requirements:
- Should know info about me, wife, children
- Should know about my computers, servers and what is running on each
- Should know my contacts, maybe from Monica
- Should update both Monica and general knowledge. All conversations need to be logged and easily audited.
- Also status – should I return to someone? A meeting was forgotten about, etc.
- Should be able to ask for data and save it. Should not be stuck waiting on data if it can’t.
What I implemented:
- Knows a lot about wife, children and me. Can recall it when needed, for example ID number for form filling.
- Knows about my computers and the system running on each. Manages the server it is on but also my web server and has very careful access to my PC (well it has SSH but I told it to be careful :)).
- Knows my friends and contacts from WhatsApp, email and Monica and updates it with our conversations.
Real world impact: Big
- Reminds me of facts when I need them, or uses them to fill forms for me without hand holding
- Lets me refresh conversations with friends and follow up when I take a peek before a meeting.
- Knows my preferences too and takes them into consideration when making decisions, for example has a heatmap of places I visit and when looking for places to buy something will look at things in my heatmap vs just close by proximity. Same for knowing I have a certain gift-card and looking for stores that accept it (as well as the cheapest option etc).
Future improvements:
- Connect it to more data sources like my bank account (it already has access to most credit cards), investment account.
- Visualize the data better
- Be proactive suggesting I do something like talk to someone if I haven’t in a while.
Trello and task mgmt
Requirements:
- Should be super integrated with Trello
- Should know my Trello system intimately
- Auditable Trello trails. Maybe give him a separate user?
- Know all of my Trello actions, including tasks I’ve randomed and then closed. Follow that closely for audit.
- Should know about my special lists like shopping lists and know where to add to them
- Pre-do tasks, sort them and also create a short “what to expect today” mail?
- Research and break tasks to small checklists
- Tips on how to do the task
What I implemented:
- Is very integrated with Trello and my process around it and special lists.
- Goes over tasks I’ve planned to work on and tries its best to do it for me or at least research as much as possible.
- Proactively pushes tasks for me to work on, especially overlooked ones.
- Adds items I tell it to the right lists, for example a book I want to read it adds to the books list, with a marker that it added it in case I need to distinguish it later.
Real world impact: Big
- The preprocessing part is not super helpful for now and it tries too hard, for example breaking into small tasks things like taking out the trash. Lots of room to improve here.
- Proactively pushing tasks is amazing since I told it to just make sure I do a small step, which sometimes is enough to push a task or to schedule time for me to actually do it. This has been great and I love the randomness.
- It also knows where I am with an Android app that sends it GPS locations and so it knows to give me appropriate tasks when I’m not home if it can (for example send someone a message, easy to do outside).
- Schedules deep work time for me and blocks distractions like X during that time.
Future improvements:
- Monitor my screen and make sure I’m on the task I’m supposed to be working on – would be a big win for documentation too
- Build my day for optimal productivity by combining calendar meetings, easy tasks and deep work.
- Estimating how much human time a task needs and following up with the actual human time it took.
- Better preprocessing of task and actually finishing a lot for me unprompted.
- This one has a lot to do and is very exciting
Separate Entity
Requirements:
- Should have his own email
- Also access to my email? Tasks from my email forwarded probably better.
- His own WhatsApp
- His own Telegram bot
- Should be added to a few groups with spouse like what to buy etc
What I implemented:
- Ability to search and read my email, with read only OAuth scope.
- His own WhatsApp, email, calendar, phone (SMS and calls still not implemented)
Real world impact: Big
- His own Telegram bot (or actually a few for multi tasking). This is my main way of talking to him.
- My email has a lot of todos and a lot of data about me in the form of 1800+ diary entries, so lots of use cases.
- Examples:
- We’re going to a restaurant we haven’t been to in a while, I asked it to recall our favourite dishes there and it did from previous orders and visits from 2023 and before.
- I’m looking to buy smart glasses that are not sold in my country. It asked two optometrists about the process and how to do it most efficiently, then found and talked with a local optometrist to schedule the exact test I need for the glasses.
Future improvements:
- Integrate him to shared groups like shopping list with wife.
- For now I approve each email and WhatsApp. Maybe I should let him have a bit more freedom and give him guardrails instead.
Research
Requirements:
- Research should use a few LLMs not only Claude
What I implemented:
- LLMs are great at research already so it came with OpenClaw.
Real world impact: Small-medium
- Good research but as expected from LLMs
Weekly introspection process and improvements
Requirements:
- Read the emails and think what to do – meta thinking – how to improve, etc
- Run /insights on Claudes
- Other similar things.
- Automatically think of and run experiments like using other LLMs, doing deep search, asking what tasks were good and what were not good.
What I implemented:
- Goes over all relevant data for my weekly introspection and finds actionable things as well as nice moments.
- Thinks of experiments to run for me.
Real world impact: Big
- I’m really surprised how well it works for the weekly. Has deep understanding and thoughts, for example catches that I don’t sleep enough, or that I should bike more as it has a big positive effect on my mood. It might seem intuitive that I should sleep more and do my hobby more, but small drifts over time can make you think you do the thing when actually you don’t.
- Experiments it suggested are great, for example cap my afternoon naps for 30 minutes to not affect my night sleep time.
- It has more personal insights that I really take into account and change my life based on (or at least collect more info to validate, like the sleep tracker).
- It is so good I now moved to a bi-weekly cadence (once in two weeks) because retrospection is so streamlined.
Future improvements:
- Adding /insights is a good idea
- Adding more sources of information like more health sensors.
- Have it be more creative and proactively seek things to track
Calendar
Requirements:
- Understand my day
- Understand how meetings affect my day
- Put meetings for Trello tasks on my calendar
What I implemented:
- Read and write access to calendar as well as to invite my wife to meetings.
Real world impact: Small
- It does a good job scheduling things like children’s vacations and reminders for me and my wife.
- The impact of this is limited
Future improvements:
- I want to plan my day exactly minute by minute including all tasks and how long they take. Yes this is a bit psychotic, but I want to experience it setting up a perfect day and me following it or just drifting off.
Misc
- Voice support – voice to voice mode or similar voice interfaces
- Implemented:
- Telegram has very meh text to speech, but usable.
- I wrote a voice to voice experiment but need to use it a bit more for verdict
- Real world impact: Small
- Future improvements:
- Basically have it listen all the time with smart glasses and when called upon to answer me quickly.
- Implemented:
- Code should be tested, linted, buildable, secure, CI
- Implemented: Mostly
- Real world impact: Small – I don’t look at the code it writes anyways
- Future improvements: ?
- Location-based alerts – understand where I am and if there is something that needs to be done on the way like groceries, etc
- Implemented: Knows where I am pretty accurately
- Real world impact: Medium
- Gives me tasks relevant to where I am if possible
- Future improvements: Actually go over my shopping lists and suggest I go to nearby places to close other todos.
- Ability to create documents in Drive or some way around it but not only MD
- Implemented: It has a “research” site that lets it use the full power of HTML and the web to embed maps, graphs, etc
- Real world impact: Medium
- Much better than MD files and Mermaid graphs
- Future improvements: ?
- Per project Claude Code with summary of last tasks and current status page. Happy – like interface (can also be multiple. Maybe Telegram? Something that can work on the phone for that)
- Implementation:
- I settled on multiple Telegram channels that are interchangeable but sometimes pinned to a project by convention
- Also have a page that shows all projects sub-agents are running on with ability for me to stop, comment, check logs etc.
- Real world impact: Small
- Future improvements: ?
- Implementation:
When I started working on this the epilogue was – time to think of new requirements, but a month or two passed and I’m already implementing wave 2 of requirements, so expect a follow up.