Glass: a desktop AI assistant that observes and organizes information in real time

Glass is an open source desktop AI assistant developed by the Pickle team. It runs in the background of the user's computer, observes screen content and audio in real time, automatically organizes information and provides instant help.Glass is different from traditional chatbots in that the user doesn't have to actively ask questions, it actively analyzes what the user is processing and provides relevant advice or knowledge. The project is hosted on GitHub, with 1.8k stars and 323 forks, and an active community of developers who are welcome to contribute code or suggest improvements. glass is currently supported on macOS, and is easy to install for those who need to efficiently organize their daily information.

Function List

• Real-time screen observation: Glass continuously monitors the content of the user's screen to analyze documents, web pages or application interfaces.
• Audio content recognition: listen to user speech or environmental audio to extract key information.
• Information organization and knowledge generation: transforming screen and audio data into structured knowledge, generating notes or recommendations.
• Running in the background: Glass works automatically in the background to provide real-time help without user-initiated actions.
• Open source support: hosted on GitHub, allowing users to view code, submit issues or contribute features.
• macOS Compatible: Designed for macOS, easy installation and lightweight operation.

Using Help

Installation process

Glass is currently only supported on macOS and requires a Node.js environment for installation. Here are the detailed installation steps:

1. Checking the Node.js version
   Open a terminal and enter the following command to check the Node.js version:

   node --version
   

   Glass requires Node.js 20.x.x. If this is not installed or the version does not match, it is recommended to use the nvm Installation:

   curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.0/install.sh | bash
   nvm install 20
   nvm use 20
   

   Once the installation is complete, run it again node --version Verify that the version is 20.x.x.

2. Cloning Glass Project
   Clone Glass's GitHub repository in Terminal:

   git clone https://github.com/pickle-com/glass.git
   

   Go to the project catalog:

   cd glass
   

3. Installation of dependencies
   Run the following command in the project directory to install the required dependencies:

   npm install
   

4. Configuring OpenAI API Keys
   Glass requires an OpenAI API key to support AI functionality. Users need to visit the official OpenAI website to get the key. Steps:
   • log in OpenAI official website, register or log in to your account.
   • Go to the API Key Management page to generate a new API key.
   • Copy the key to a configuration file in the Glass project directory (refer to the GitHub repository for instructions on how to do this, usually the .env (Documentation).
     For a more detailed configuration tutorial, refer to the video guide provided by the Pickle team. [](https://github.com/pickle-com/glass)
5. Run Glass
   After the configuration is complete, start Glass by running the following command in the project directory:

   npm start
   

   Glass will launch on the macOS desktop and will continue to work in the background once it is running.

Key Features of Using Glass

The core function of Glass is to observe the user's screen and audio in real time and provide intelligent help. Below is the detailed operating procedure:

1. Real-time on-screen observation
   When Glass is activated, it automatically monitors the content of the screen. For example, if a user is editing a document or browsing a web page, Glass analyzes the text, images, or tables on the page to extract key information.
   • Operating MethodsGlass runs automatically upon startup: no additional setup is required. When the user opens any application (e.g. browser, Word or code editor), Glass displays a small window in the bottom right corner showing the extracted information or suggestions.
   • typical exampleWhen a user writes an article about AI, Glass detects the keyword "artificial intelligence" and automatically provides a definition or a link to the latest research.
2. Audio Content Recognition
   Glass listens to the user's voice or ambient audio, making it ideal for meeting recording or voice note-taking scenarios.
   • Operating Methods: Make sure the microphone is connected and authorize Glass access. When a user speaks or plays audio in a meeting, Glass converts the speech to text and extracts highlights.
   • typical example: Discuss the project plan in a Zoom meeting, and Glass generates a summary of the meeting, listing tasks and deadlines.
3. Information organization and knowledge generation
   Glass integrates screen and audio data into structured knowledge, generating notes, task lists or suggestions.
   • Operating Methods: Without the user having to do anything manually, Glass organizes the information in the background and displays it in a small window. Clicking on a note in the window allows you to save it as a text file or export it to other tools (e.g. Notion).
   • typical example: While the user is browsing the web, Glass detects a technical article and generates notes containing the title, abstract and key points.
4. Back-office operations and real-time help
   Glass is designed to be a lightweight tool that uses few resources and is suitable for long runs.
   • Operating Methods: After startup, Glass remains in the background and can be accessed via a shortcut key (default). Cmd + G) to bring up or hide the window.
   • typical example: The user encounters an error while programming, Glass detects the error code and suggests a possible solution or provides a Stack Overflow link.
5. Community Contributions and Feedback on Issues
   Glass is an open source project, and users can participate in development or provide feedback via GitHub.
   • Operating Methods: Access GitHub RepositoriesYou can submit bugs or feature suggestions by clicking on the "Issues" tab. Users can also fork the repository, modify the code and submit a pull request.
   • typical example: If you find that Glass is displaying abnormally on high-resolution screens, you can create an issue on GitHub describing the problem and attaching screenshots.

caveat

• private business: Glass requires access to the screen and audio, and users are advised to read carefully the Pickle Privacy Policy Understand how data is processed.
• system requirements: Currently only macOS is supported, Windows and Linux versions have not yet been released.
• Community Support: Join Pickle's Discord Community Or follow their X account for updates.

application scenario

1. proceedings
   Glass is for users who need to record meetings efficiently. It listens to meeting audio in real time, generates summaries and task lists, and reduces time spent taking manual notes.
   For example, if a product manager discusses development progress in a daily stand-up meeting, Glass automatically generates a table with topics, leads, and deadlines.
2. Study and Research
   As a student or researcher browses academic web pages or literature, Glass extracts key information and generates notes or reference lists.
   For example, reading a paper on machine learning, Glass organizes the core formulas and conclusions for easy review.
3. Programming Aids
   As developers write code, Glass can detect errors or provide suggestions for code optimization.
   For example, when Python reports an error, Glass analyzes the error message and recommends a fix.
4. Daily task management
   Glass helps users organize information for daily tasks, such as extracting to-do lists from emails or documents.
   For example, if a user receives an email with a meeting schedule, Glass automatically generates a calendar reminder.

QA

1. What systems does Glass support?
   Currently only macOS is supported, Windows and Linux versions are in development.
2. How is privacy protected?
   Glass requires screen and audio permissions, and data is processed in accordance with the Pickle Privacy Policy. Users can suspend or disable permissions at any time.
3. Do I need an OpenAI account?
   Yes, Glass needs to be configured with an OpenAI API key to support AI functionality. Users can get it through the official OpenAI website.
4. How can I get involved in Glass development?
   interviews GitHub RepositoriesThe community is open to any contributions.