Apple Introduces New API to Enhance Siri's Content Awareness in iOS 18.2 Beta

Apple has recently released the iOS 18.2 beta 2 to developers and testers, marking a significant step toward the next iteration of its mobile operating system. Scheduled to be available in early December, this update promises a variety of new features harnessing Apple Intelligence. Among these updates is a novel application programming interface (API) aimed at enabling developers to enhance content awareness by utilizing Siri and Apple Intelligence, allowing them to relay information from on-screen content to external services for further processing.

Apple Unveils API to Enhance Siri's Content Awareness

The documentation for the new API, found on the Apple Developer portal, is titled 'Making onscreen content available to Siri and Apple Intelligence.' It is specifically crafted to facilitate access to the on-screen elements of an application. This capability empowers both Siri and Apple Intelligence to comprehend what the user is currently interacting with on their device. With the implementation of the onscreen content API, developers can enable their applications to relay the visible content to Siri or Apple Intelligence when a user actively prompts for assistance. This feature allows the information displayed on the user’s screen to be transmitted to third-party services such as OpenAI's ChatGPT for processing. Apple has illustrated this feature using a scenario where a user browsing the internet can engage Siri by saying, “Hey Siri, what's this document about?” This command would prompt Siri to generate a summary of the document currently being viewed.

Potential Applications for the New API

Developers have the opportunity to incorporate onscreen awareness across a variety of applications, including web browsers, document readers, file management systems, email clients, photo applications, presentation tools, spreadsheet software, and word processors. Apple has indicated that this list is not comprehensive, suggesting that more types of applications may leverage this API as it evolves. It is important to note that iOS 18.2 will not introduce the anticipated update to Siri, which promises substantial improvements in functionality. This upgrade is set to debut with iOS 18.4, alongside support for in-app actions that Apple intends to roll out in April 2025. This timeline gives developers ample opportunity to integrate support for the new API into their applications.