Starting in iOS 14 and macOS Big Sur, developers will be able to add the capability to detect human body and hand poses in photos and videos to their apps using Apple's updated Vision framework, as explained in this WWDC 2020 session.
This functionality will allow apps to analyze the poses, movements, and gestures of people, enabling a wide variety of potential features. Apple provides some examples, including a fitness app that could automatically track the exercise a user performs, a safety-training app that could help employees use correct ergonomics, and a media-editing app that could find photos or videos based on pose similarity.
Hand pose detection in particular promises to deliver a new form of interaction with apps. Apple's demonstration showed a person holding their thumb and index finger together and then being able to draw in an iPhone app without touching the display.
Additionally, apps could use the framework to overlay emoji or graphics on a user's hands that mirror the specific gesture, such as a peace sign.
Another example is a camera app that automatically triggers photo capture when it detects the user making a specific hand gesture in the air.
The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing. The algorithm can also experience difficulties if a person is close to edge of the screen or partially obstructed.
Similar functionality is already available through ARKit, but it is limited to augmented reality sessions and only works with the rear-facing camera on compatible iPhone and iPad models. With the updated Vision framework, developers have many more possibilities.
Saturday August 10, 2024 5:00 am PDT by Tim Hardwick
Apple typically releases its new iPhone series in the fall, and a possible September 10 announcement date has been floated this year, which means we are just one month away from the launch of the iPhone 16. Like the iPhone 15 series, this year's lineup is expected to stick with four models – iPhone 16, iPhone 16 Plus, iPhone 16 Pro, and iPhone 16 Pro Max – although there are plenty of design...
Apple's M3 MacBook Pro is seeing multiple high value discounts on Best Buy and Amazon today, with up to $1,000 off select models. This includes a new all-time low price on the entry-level M3 512GB 14-inch MacBook Pro at $1,299.00, down from $1,599.00, and a massive $1,000 discount on the high-end 16-inch model exclusively for Best Buy members. Note: MacRumors is an affiliate partner with Best...
Tuesday August 13, 2024 4:01 pm PDT by Juli Clover
Multiple rumors have suggested that the iPhone 16 models are going to have an all-new button that's designed to make it easier to capture photos when the devices are held in landscape mode. Apple calls the button the Capture Button internally, and it is going to be one of the most advanced buttons that's been introduced to date with support for multiple gestures and the ability to respond to ...
Apple is beta testing iOS 18 and the first update to iOS 18 concurrently, and we got the second betas of iOS 18.1, iPadOS 18.1, and macOS Sequoia 15.1 today alongside the sixth betas of iOS 18, iPadOS 18, and macOS Sequoia 15. Many of the changes in iOS 18.1 are focused on bringing the .1 betas in line with the standard betas, which recently received updates to Photos and Safari, while...
Thursday August 8, 2024 4:40 am PDT by Tim Hardwick
Apple's iPhone development roadmap runs several years into the future and the company is continually working with suppliers on several successive iPhone models simultaneously, which is why we sometimes get rumored feature leaks so far ahead of launch. The iPhone 17 series is no different – already we have some idea of what to expect from Apple's 2025 smartphone lineup. If you plan to skip...
Apple's rumored iPhone 17 "Slim" could be positioned as an iPhone "Air" to boost sales, according to Bloomberg's Mark Gurman. In the latest edition of his "Power On" newsletter, Gurman explained how the "fourth" model in the iPhone lineup since 2020 (the iPhone 12 mini, iPhone 13 mini, iPhone 14 Plus, and iPhone 15 Plus) has largely been a commercial failure. In the case of the Plus model,...
Honestly, this seems like the kinda stuff that'd make Apple AR compelling—being able to draw in midair means you’d also be able to navigate an interface in midair with just your hands.
Using AR/VR without bringing a controller everywhere seems analogous to what set the iPhone apart from other touchscreen phones in 2007; you didn’t need a stylus.
The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing.
I don't think the wizarding community is going to be too happy about this...
