Bumblebee

Desktop automation with a learned mouse policy

By Arpan Bhandari

Source: https://arpan.sh/projects/bumblebee

Mouse and keyboard automation for Python, with an optional motion policy trained on recorded cursor movement.

Bumblebee combines Python desktop automation with a mouse policy trained on real cursor recordings. I built the data preparation and training workflow, then integrated the learned motion into a mouse and keyboard library.

The challenge was making recorded movement useful beyond the exact screen positions where it was captured.


Learning movement rather than coordinates

I normalized each cleaned trace relative to its start and destination. That preserves the shape and speed of a movement while allowing it to be reused between different points on the screen. Pauses, invalid timestamps, and large jumps are filtered before training.

For the learned policy, I used Soft Actor-Critic. Reaching the target comes first in the reward design; matching the character of a recorded path adds a further reward. Reversing that priority could produce convincing movement that fails to reach the place the caller requested.

I also kept endpoint correction in the mouse controller. The policy chooses a path, while the controller remains responsible for finishing at the requested coordinate. That makes the package usable without treating every model prediction as exact.


Making the model optional

A mouse automation library should be useful without installing a training environment. I separated the motion policy from input execution and made the ML dependencies optional. Applications can use the ordinary movement profiles or load the packaged policy through the same mouse API.

That separation also makes it possible to replace the model without changing the rest of an automation script. The trained policy ships with the package, while training data and larger artifacts stay separate.

Bumblebee is available on PyPI [https://pypi.org/project/the-bumblebee/]. The model card [https://github.com/arpan404/bumblebee/blob/master/src/bumblebee/models/MODEL_CARD.md] records its training setup and limits.