AI can use your computer

Demo on how to apply Anthropic's new feature today

Martin Musiol
October 23, 2024 • Estimated Reading Time: 7 minutes

In the screen capture below, you can see how Claude AI can steer your computer per mouse and keyboard. 🤯

Now, there are no limits. AI can do anything you can! Not quite yet, but it’ll come.

Also, spend another minute to see Perplexity AI’s new reasoning feature.

We’re using an AI tool that creates how-to videos, so we don’t repeat the same thing over and over (carefully selected sponsor)

The image is an advertisement promoting the use of AI to create how-to videos 11x faster. The design features a vibrant gradient background, with the following key elements highlighted: 1. **Headline**: "Create how-to videos 11x faster with AI" 2. **Key Features**: - **Magic Capture**: Suggests an AI-driven tool that can capture processes efficiently. - **AI-generated voiceover**: Indicates automatic voiceover creation for videos. - **Editor**: Refers to an AI-powered video editing tool. - **Smart Sharing**: Implies easy sharing options for the created videos. At the bottom, a blue button says "Try For Free," suggesting a free trial of the product. This content is ideal for companies or creators looking for tools to streamline video creation processes using AI.

Your team + this AI video tool = creating how-to videos once and for all

For all training use cases: new hires, customer onboarding, product documentation
For all languages: it translates the video into 40+ languages
For all platforms: integrated with Salesforce, Notion, Zendesk…
For all formats: can share as link, mp4, Pdf, Html

No more unnecessary meetings, no more wasting time, no more low-engagement training content.

Just, add the extension, create a free account, record and the AI will: generate a step-by-step storyline, include highlights, add voiceover, produce a ready to share video.

Computer Use - The breakthrough feature

Anthropic has released a new feature called computer use. It is a breakthrough feature because their top model can now use your computer’s mouse and keyboard to work with your computer - as if it is you.

Currently, it is still a bit buggy, but we know how AI gets exponentially better.

In the following video, I show some of its capabilities and how to set it up.

The second image appears to be a screenshot of a webpage introducing new features for a product called **Claude 3.5 Sonnet** and **Claude 3.5 Haiku**, published on October 22, 2024. The text on the webpage describes the upgrades to these models and indicates a 5-minute read. The visuals include a stylized drawing of a hand pointing at a large cursor, suggesting interaction with a computer or digital tool. In the bottom-right corner of the screenshot, a video is playing, with a person speaking, likely explaining the content of the article. The text below the image begins with a brief introduction of the upgraded models. This screenshot likely pertains to a new release or update from Anthropic, as the website's domain ("anthropic.com") is visible in the browser. The update seems focused on improvements to AI models related to computer use, potentially geared towards enhancing usability or performance. Let me know if you need further assistance with this or another aspect!

Click on image or here: https://www.loom.com/share/52b990c0839a4db587feba8a8658a015?sid=85a9f633-5695-4382-bf41-787d871ab004

Ok, that was me using the pre-build demo of Anthropic. They use a Docker enviroment to provide a save way demoing its capabilities.

Here are two more examples from the team.

1) Planning out a trip + scheduling

2) Computer use for coding

Why is it a breakthrough feature?

Now, there are no limits for AI to work on your behalf.

In addition to their tool use feature, which is for calling APIs to perform tasks like calling SLMs (small language models) to run a sub-task, it can now use your mouse and keyboard to accomplish anything you would be able to - except it has vast knowledge.

Implement it yourself! With AI at hand, it is relatively easy. I certainly will.

From Claude’s documentation:

This image contains a code snippet written in Python, showing an interaction with the Anthropic API. The script is leveraging the Claude 3.5 Sonnet model, specifically a version labeled claude-3-5-sonnet-20241022. Here's a breakdown of the code: The Anthropic client is being instantiated with client=anthropic.Anthropic(). A message creation function (client.beta.messages.create) is used to generate a response using the Claude 3.5 Sonnet model. The parameters used include: max_tokens=1024: This limits the response token size. A set of tools is defined: A computer tool (computer_20241022) with a display size of 1024x768 pixels. A text editor tool named str_replace_editor. A bash tool named bash. A message is passed with the user role and content: "Save a picture of a cat to my desktop." The beta feature used is labeled "computer-use-2024-10-22". Finally, the response is printed, suggesting that this script executes a task within a simulated desktop environment using AI tools like a text editor, bash terminal, and a virtual display. It showcases how AI models like Claude 3.5 Sonnet can interact with simulated tools to perform tasks, possibly involving automation or advanced task orchestration through API calls.

https://docs.anthropic.com/en/docs/build-with-claude/computer-use

👆️ If you have built something nice, I happy to feature it in the newsletter.

Oh, and also, as a developer, I agree:

Claude 3.5 sonnet (new)
WHY CANT ANYONE NAME THINGS PROPERLY!?!?
CALL IT 3.6 or 3.5.1
OR SOMETHING
— Nick Dobos (@NickADobos)
4:20 PM • Oct 22, 2024

Would you be interested in a premium version of this newsletter?

No ads, and more in-depth content that you can co-shape.

Perplexity AI has now reasoning models in the backend to answer more complex tasks

Perplexity Pro is transitioning to a reasoning powered search agent for harder queries that involve several minutes of browsing and workflows. Check it out! It’s still in beta and has some issues but we’re going to keep making it better!
— Aravind Srinivas (@AravSrinivas)
7:00 PM • Oct 22, 2024

It’s time to let AI create how-to videos for your team

Guidde showing how meetings can be replaced.

That’s a wrap! I hope you enjoyed it.

Martin

Do you write newsletters? I use Beehiiv and highly recommend it.
AI for your org: We build custom AI solutions half the market price, and time (building w/ AI Agents). Contact us to know more.
Would you like to sponsor a post?
My book - Generative AI: Navigating the Course to AGI.
Generativeai.net