Full Video:
Functionality & Usage
What exactly can I do with ChatGPT integrated into my Apple devices that I couldn’t do before (or was much harder to do)?
OpenAI’s integration brings three major capabilities: direct Siri handoff to ChatGPT for complex queries, native document analysis and composition through Apple Intelligence, and visual analysis through iPhone’s camera control. Sam Altman and the team demonstrated how users can seamlessly ask Siri to delegate tasks to ChatGPT, analyze 49-page PDFs directly in MacOS, and use visual intelligence to analyze real-world scenes through the camera. This integration allows ChatGPT to be accessed from virtually any application on Mac, iPhone, and iPad without switching apps.
Can I use all ChatGPT features (including plugins and custom GPTs, if applicable) through this integration?
Based on the demonstration, the integration appears to offer core ChatGPT functionalities like text generation, image analysis, and document processing, but there was no mention of plugins or custom GPTs. Dave Cummings and Miqdad Jaffer showed basic conversational features and image generation, but advanced features weren’t demonstrated or discussed.
How is this different from just using the regular ChatGPT app on my Apple devices?
This Apple Intelligence integration allows ChatGPT to be accessed system-wide through Siri, writing tools, and camera controls without opening the ChatGPT app. Users can seamlessly transition conversations from Siri or Apple Intelligence to the full ChatGPT app by clicking the ChatGPT button at the bottom of interactions. The integration provides native document handling capabilities directly within Apple’s ecosystem.
Does this integration work offline?
The transcript doesn’t address offline functionality, and since ChatGPT requires internet connectivity for processing requests, it’s likely that an internet connection is required for the integration to work.
Will using ChatGPT through Apple Intelligence count towards my ChatGPT usage limits (if I have a paid subscription)?
The transcript doesn’t specifically address usage limits or how the integration affects ChatGPT subscriptions, though it mentions that users can use the integration without an account but will get enhanced features with one.
Can I access my previous ChatGPT conversations through this integration?
OpenAI’s integration allows users to continue conversations from Apple Intelligence directly in the ChatGPT app by clicking the ChatGPT button at the bottom of interactions. Dave Cummings demonstrated how conversations seamlessly transfer to the desktop app, where they appear in the sidebar for future reference.
What types of requests are best suited for invoking ChatGPT through Siri?
According to Sam Altman and Dave Cummings, complex tasks that go beyond Siri’s basic capabilities are automatically handed off to ChatGPT. ChatGPT integration excels at tasks like party planning, document analysis, and complex queries that require deeper understanding and reasoning.
Are there specific commands or phrases I need to use to ensure Siri hands off to ChatGPT correctly?
Miqdad Jaffer demonstrated that users can directly say “ask ChatGPT” to Siri for an immediate handoff to ChatGPT. Siri will also automatically suggest ChatGPT handoff when it recognizes a complex task that could benefit from ChatGPT’s capabilities.
How can I tell if Siri is going to use Apple’s internal processing or send my request to ChatGPT?
Siri displays a confirmation prompt when it decides to hand off to ChatGPT, showing “Working with ChatGPT” on the screen. Users maintain control over what information is shared with ChatGPT through these confirmation prompts.
Can I adjust the level of confirmation required before Siri sends a request to ChatGPT?
OpenAI’s integration includes a toggle in Settings under Apple Intelligence and Siri to enable or disable confirmation for ChatGPT requests, allowing users to control when and how their information is shared with ChatGPT.
Can I use this Siri integration with voice commands only, or is typing required in some cases?
Based on the demonstration, the integration works with both voice and text input – Dave Cummings specifically showed how Mac users can type to Siri by double-tapping the command key, though this was mainly to prevent multiple devices from responding during the demo. Voice commands appear to be fully supported on all devices.
In which Apple applications will the ChatGPT writing tools be available?
Dave Cummings demonstrated that ChatGPT writing tools work through Apple Intelligence across virtually any application on MacOS, including Preview for PDF handling. OpenAI’s integration allows users to invoke writing tools from any application through Siri commands or system shortcuts.
What kind of document composition can ChatGPT handle (e.g., emails, essays, code)?
While the demo focused primarily on PDF analysis (showing a 49-page system card document), the specific document types weren’t extensively detailed in the presentation. Dave Cummings mentioned that the writing tools can both refine existing documents and compose new ones from scratch.
Can I provide specific instructions or prompts within the writing tools interface?
Based on Dave’s demonstration, users can provide specific instructions through Siri or typing interface, as shown when he requested analysis of coding techniques and later asked for a pie chart visualization. The system allows for follow-up questions and specific formatting requests.
How does the summarization feature compare to Apple’s existing summarization capabilities?
The demonstration didn’t provide a direct comparison between ChatGPT’s and Apple’s native summarization features. However, Dave Cummings highlighted that ChatGPT can handle complex tasks like analyzing a 49-page technical document and creating visual representations of the content.
Can I refine or edit the ChatGPT-generated content directly within the writing tool?
The presentation showed that users can continue conversations and refine content by clicking the ChatGPT button at the bottom of interactions, which opens the full ChatGPT app with the conversation history. Dave demonstrated this workflow when analyzing the technical document and requesting additional visualizations.
On which devices and with which apps will the visual intelligence feature work?
Miqdad Jaffer demonstrated that visual intelligence works on iPhone through the camera control button, specifically mentioning iPhone 16. Users can access this feature by long-pressing the camera control button to activate visual intelligence capabilities with ChatGPT.
How accurate is the object recognition and analysis through the camera?
The demonstration showed ChatGPT accurately identifying people in the frame, their positions (left, middle, right), and details about their clothing like Christmas sweater patterns. The system was able to make comparative analyses between multiple subjects in the frame.
What kind of questions can I ask about the things my camera sees? Are there limitations?
During the demo, Miqdad showed that users can ask for identification of objects, comparative analysis (like ranking Christmas sweaters), and descriptive details about what’s in frame. ChatGPT could analyze clothing details and make subjective assessments based on visual information.
Can I use visual intelligence with photos and videos already in my library, or is it only for real-time camera views?
The demonstration only showed real-time camera functionality through the camera control button, and there was no mention of using the feature with existing photos or videos from the library. This specific capability wasn’t addressed in the presentation.
Setup & Access
When exactly will this integration be available on my iPhone, iPad, and Mac? (Specific OS versions needed)
OpenAI announced this integration as Day 5 of their “12 Days of OpenAI” initiative, with Dave Cummings specifically mentioning it’s available in MacOS 15.2. The integration works on iPhone, iPad, and Mac devices, though specific iOS version requirements weren’t explicitly stated.
Is this rolling out in phases, or will it be available to everyone at the same time?
The transcript doesn’t explicitly address the rollout strategy, though Sam Altman presents it as a current release saying “our friends at Apple are releasing the ChatGPT integration.”
Do I need to update my device’s operating system to a specific version to use this feature?
Dave Cummings mentions MacOS 15.2 is required for the ChatGPT extension, marking this as a new feature in that OS version. iOS version requirements weren’t specifically mentioned in the presentation.
What are the exact steps to enable Apple Intelligence and the ChatGPT extension in my settings?
Miqdad Jaffer demonstrated the setup process: Go to Settings > Apple Intelligence and Siri > Enable Apple Intelligence > Scroll down to find ChatGPT extension > Enable ChatGPT > Optionally log into account > Choose whether to confirm ChatGPT requests.
Do I need an existing OpenAI/ChatGPT account to use this integration, or can I use it anonymously for some features?
Sam Altman explicitly states users can use the integration without an account, though having one provides enhanced features. The integration supports both anonymous and authenticated usage.
What are the benefits of linking my ChatGPT account versus using it anonymously?
While the presentation mentions that “if you have an account it’ll be even better,” specific enhanced features for account holders weren’t detailed in the demonstration.
Can I switch between using my account and using it anonymously?
The transcript doesn’t explicitly address account switching capabilities, though Dave mentions users are “in control” of what they share with ChatGPT and can use it “fully anonymous or with an account.”
Cost & Availability
Is the integration of ChatGPT through Apple Intelligence free to use?
The transcript doesn’t explicitly address pricing structure for the Apple Intelligence integration with ChatGPT. While Sam Altman mentions it can be used without an account, no specific cost details were provided.
Are there any limitations to the free usage, and will there be paid tiers or premium features in the future?
The presentation doesn’t outline any usage limitations or mention future paid tiers. Sam Altman and Dave Cummings only mention that using the integration with an account provides enhanced features, without specifying what those enhancements are.
If I’m a ChatGPT Plus subscriber, do I get any enhanced benefits through this integration?
While Sam Altman mentions that having a ChatGPT account will provide better features, the specific benefits for ChatGPT Plus subscribers weren’t addressed in the demonstration.
On which specific iPhone, iPad, and Mac models will this integration work? Are there minimum hardware requirements?
Miqdad Jaffer mentions “iPhone 16” for the camera control feature, and Dave Cummings specifically notes MacOS 15.2 is required. Beyond these specific mentions, the presentation doesn’t detail minimum hardware requirements or provide a comprehensive list of compatible devices.
Troubleshooting & Limitations
What should I do if ChatGPT doesn’t respond or provides an error through the Apple integration?
The presentation doesn’t address troubleshooting procedures or error handling. Neither Sam Altman nor the engineering team discussed what to do when issues arise with the integration.
What if Siri misunderstands my request and doesn’t hand off to ChatGPT correctly?
Dave Cummings mentioned that Siri will ask for confirmation before sending requests to ChatGPT, giving users control over what information is shared. Users can presumably decline incorrect handoffs and rephrase their requests.
Where can I find help and support for this integration?
The demonstration doesn’t specify support channels or resources for troubleshooting the Apple Intelligence integration with ChatGPT.
Are there any tasks or types of information that ChatGPT through Apple Intelligence cannot handle?
The demonstration showcases specific capabilities like document analysis (up to 49 pages), image generation, and visual intelligence, but doesn’t outline specific limitations or restrictions of the integration.
Are there any known limitations or bugs in this initial release?
No specific limitations or bugs were mentioned during the presentation, though Dave Cummings jokingly suggested “filing a bug” during the Christmas sweater contest, indicating there’s a system for tracking and addressing issues.
Will the functionality and features evolve over time? How often can we expect updates?
Sam Altman briefly mentions that “faster image gen is coming,” suggesting ongoing development, but no specific timeline or update schedule was provided for future enhancements.
Full Transcript
ChatGPT x Apple Intelligence—12 Days of OpenAI: Day 5
(00:00) [Music] Welcome to Day 5! Today is about making ChatGPT even easier to use. Our friends at Apple are releasing the ChatGPT integration in iOS for iPhone and iPad, and also on macOS. We really want to make ChatGPT as frictionless and easy to use everywhere. We love Apple devices, and so this integration is one that we’re very, very proud of. You can use it without an account; if you have an account, it’ll be even better. And I think we should just jump right in and show you how it works.
(00:35) Yeah, cool. So, this is Dave. I’m Dave. I’m from the engineering team. Um, as Sam says, we love Apple devices. I’ve been using one for 30 years… on lots of Apple devices.
(00:36) Me too! On lots of Apple devices now. Uh, you will be able to invoke ChatGPT directly from the operating system. So this will work on iPhone, on iPad, and on Mac. Uh, we’ve got three different integrations starting today, and we’re going to show you all of them today. Um, the first is Siri. So when Siri thinks that it would be helped by giving a task over to ChatGPT, it can just hand it off. The next is writing tools. So writing tools in Apple Intelligence lets you refine documents and summarize them, get key points, but now you can also use ChatGPT uh to compose a document from scratch, which is super cool. Um, and also, uh, we’ve got camera control on the iPhone 14, which lets you invoke visual intelligence where you can use ChatGPT uh to learn more about the thing that you’re looking at, which is super cool. Um, so this is Miqdad from the product team, who’s going to show us some of this stuff.
(01:40) All right, let’s uh, let’s open up our iPhones a little bit. So now that I have this thing up, there’s a couple things that you want to pay attention to once you get the iPhone. The first thing is that you have to enable Apple Intelligence. So we’ll go into settings and we will go right into Apple Intelligence and Siri. I’ll enable Apple Intelligence, which I have already. And then, as I scroll down, the new ChatGPT extension is now available. Cool. That says “Chat.” When we open it up, you’re going to be able to enable ChatGPT, log into your account—in this case, I’m already logged in—and then there’s also this option for you to confirm ChatGPT requests. So anytime, as Dave mentioned, Siri thinks, “Hey, this is a good idea to ask ChatGPT,” uh, it will prompt you to do so before you send your information over. And then you’ll also have access to the app, which you can update, install, or open directly from here. But I’ve got this all set up, so let’s go back into our main screen and let’s… I mean, I feel like we’re in a festive mood here; something’s going on. Uh, so let’s maybe get a Christmas party organized. And there’s a quick shortcut if you want to go directly to ChatGPT, you can just ask Siri to ask ChatGPT. So I’m going to invoke Siri—new rainbow thing shows up—uh, “Can you ask ChatGPT to organize a Christmas party for us?” So right away we see the “Working with ChatGPT.” It’s going to go back and forth, and I’m going to get a response. In this case, I see a whole bunch of different things: guest list, music, and entertainment, man.
(02:40) Sam, any follow-up? Can… can we ask you for that holiday playlist?
(02:41) Yeah, let’s do that. Can we get a holiday playlist made? And I don’t know about you guys, but there is one song I’m specifically looking for, and it is… it’s number one. If Mariah Carey weren’t there, I think we’d have to file a bug.
(03:12) Yeah, I… I think this is a good solid list. Uh, I like it. I want to be able to like edit more in the ChatGPT app. So there’s this like blue icon at the bottom that says ChatGPT. If I click on that, it’s going to open up ChatGPT with all of my conversation already integrated. So cool. And I can do the normal things of saying, hey, like, uh, “Add emoji to my playlist,” and it’ll do that. But I… I think what we need is a little bit more of a festive thing. So I’m hoping this is festive, but let’s make an album art cover. Uh, and for some reason, I don’t know why, but I’m feeling as though a frog would really make it pop.
(03:46) I’m having the same feeling too. Yeah, I don’t know. It’s something in the background, but I feel like a frog… include a frog in the cover. Do you think ChatGPT knows what a frog is? Uh, surely not. Let’s see what comes through. And uh, we’ve got ourselves a beautiful… We do have faster image gen coming. Always great to see this come through. Hey, here we go! Frog! Giant happy frog in the middle with holiday vibes. I think we’ve got the makings of a party. And I, as we’re in already a festive mood, I’d love to like get the visual intelligence part of this rolling. Um, I think we should have a Christmas sweater contest, and I think ChatGPT should help us rank who’s most to least fun. Um, in preparation for this, I feel like the right move is to let Santa dance. So if you’re bringing a name for the sweater contest… I will do this. I feel a little outgunned here. Do anything… So I’m going to long-press on the camera control button, and I’ve got uh, the visual intelligence set up, and now we can see our lovely camera crew. Yeah, we’re going to break the fourth wall for a second. Can everybody wave? Hey, guys! So let’s zoom in on our little picture here, and I’m going to press this “Ask” button, and that’ll ask ChatGPT directly. First thing it’s going to do is it’s just going to like identify some of the things going on in the picture. My poor Santa’s got covered, so we’ll see how this goes. Um, we’re having a Christmas sweater contest: Miqdad on the left, Dave in the middle, and Sam on the right. Uh, rank… judge… judge… rank us…
(05:06) We… we don’t judge. We listen, but we don’t judge. Uh, rank us from most to least fun. All right, moment of truth here. Sam wins! That… file a bug for that. So I’m last. The sweater features a more traditional design with a subtle festive pattern… What? Hang on. Scroll down. Wait, wait, wait. All the sweaters have their charm, but Sam stands out as the most fun with its bold… Let me tell you something. Someone bought like three options for me. I had to just pick right before this which one I was going to wear, and I thought that if I was going to win, it was because it was different. And you know… Let’s make a trophy image for Sam. I… I… I was not expecting this result. I really thought the… I thought… I thought you guys were going to… I thought… I thought you guys were going to… Congratulations, sweater champion! That’s a cool… great. Yeah, love that. Okay, so… so we’ve got the fun part of it taken care of. Uh, when out of my sales, and I’m going to move Santa into a slow down. I just want to say I feel like that was very unjust. I appreciate you, Sam. Uh, but we’ll pass to Dave now, who’s going to show us how we can use Apple Intelligence to get back to work.
(07:03) Yeah, so fun and games are over. Um, if I’m back at my Mac… just like Miqdad showed earlier with the iPhone, if you go into system settings, you can enable Apple Intelligence. And so… uh… right here we’ve got Apple Intelligence enabled, and new in macOS 13.2, that we didn’t have before, is the ChatGPT extension. So just like on the iPhone or iPad, you can use this fully anonymously or with an account. We’ve got here an account that we have linked. Um, and what’s really cool is now that I’ve enabled Apple Intelligence, I can invoke Siri or writing tools from just about any application, and it’s super easy to invoke Siri. So I’ve disabled “Hey Siri” so that all of the MacBooks and iPhones in this room don’t go off at once. Um, but you can type to Siri now. So if you either click on the little Siri icon on the top right or hit the uh, Command key twice, it brings up the little Siri prompt. That’s really cool. And again, like anytime Siri thinks, “Ooh, this is a complex task; maybe I should get ChatGPT’s help with it,” it’ll hand it off. It’ll ask for confirmation, um, but then get uh, the help of our super powerful models to make it happen. So what are some places where we might use this? Well, um, gosh, in catching up on all of uh, our launches this week, uh, I’ve got here the… uh… system card… nine-page system… 49-page PDF. So I’ve opened this in Preview, but again, like in any application, I can just double-tap the Command key to bring up “Type to Siri.” Uh, how about this… this doc is super long. Um, how did they get the model to be so good at coding? Oh, I spelled “model” wrong; that’s why. And you know what? I wasn’t in the system card, so let me make sure we give the right credit here. So what happens is Siri recognizes, “Okay, this is a complex task; let’s send it off to ChatGPT.” Um, but before we do, it wants to know, do we want to send any of the content that I’m looking at as part of the request? And we can either send the screenshot or the full 49-page PDF. Send it all… send it all. At all times, I’m in control of what I share with ChatGPT. And so now we’re uploading this whole document for ChatGPT to process along with uh, my query of like, “How in the world did they get this model so good at coding?” Um, and awesome! There’s the answer. Maybe this saves me a good bit of time. Miqdad showed earlier on the iPhone that if you tap this little ChatGPT button down at the bottom, it will open this in ChatGPT, and that’s super cool because then I get to save it for later. I get to continue the conversation. I get to do more with it. So like one of the cool things I might do now that I have it saved here… and it shows up in my sidebar over there… is… uh… okay, as a side note, the ChatGPT desktop app is so good now. If you try… so good. It’s so good. Uh, I’m a more visual learner. Can we make a pie chart of the respective influence of each of the techniques on making it good at coding? And so this is just picking up where we left off. It already has the full PDF. It’s able to read through it again, analyze it, um, reason about uh, how uh, I guess like the relative ranks and importance of uh, each of the different techniques here, and ultimately write some code to help visualize it. Cool. Um, and I’ll check with our colleagues over on the reasoning team if this is representative, but this helps me better understand the paper, which is super cool. Um, so all of this from just about any application on your Mac… with you everywhere… in your iPhone in your pocket. Um, and uh, we’re hoping that a lot of folks who maybe are already using ChatGPT are now getting to use it uh, in more places with uh, less friction.
(10:41) Yeah, so we’re thrilled about this launch… or this release. This button makes it really easy to talk to ChatGPT. This button makes it easy for ChatGPT to look at something for you. We hope you love it. Uh, grateful to our friends at Apple, and have a good day.
