Deep Fusion в камере iPhone 11 и 11 Pro: что это, как работает и как включить?
На презентации iPhone 11 и iPhone 11 Pro компания Apple поведала о появлении новой технологии Deep Fusion. Вице-президент по маркетингу, Фил Шиллер, назвал ее «безумной наукой вычислительной фотографии». Но оценить Deep Fusion в действии пользователи смогли только с выходом iOS 13.2.
Что делает Deep Fusion при съемке фото в iPhone?
Новшество предназначено для автоматического создания и улучшения снимков. Принцип действия этой технологии чем-то напоминает HDR. Камера iPhone создает несколько снимков с разной экспозицией и комбинирует их для обеспечения максимальной детализации. Участие в обработке принимает нейросеть. Но если HDR предназначена для расширения динамического диапазона и повышения детализации в тенях, то Deep Fusion позволяет улучшить общую детализацию снимков.
После нажатия и до срабатывания кнопки затвора камера самостоятельно делает три снимка, установив для них короткую выдержку. А уже после первого этапа съемки – еще 4 кадра, один из них с длинной выдержкой. Все эти 7 снимков обрабатываются попиксельно и объединяются на основе лучших вариантов. Такая работа позволяет увеличить детализацию, убрать шумы, засветы и перетемненные участки. На выходе получается максимально бесшумный и четкий снимок.
Как включить автоматическое улучшение фотографии Deep Fusion на iPhone?
Чтобы функция Deep Fusion работала, перейдите по пути Настройки → Основные → Камера и отключите опцию «Захват фото вне кадра». Впоследствии функция автоматически будет применяться аппаратом во время съемки в зависимости от условий.
Блогер Тайлер Стейльман одним из первых протестировал работу Deep Fusion на своем iPhone 11. Объектом для съемки он выбрал самого себя. Для сравнения блогер привел фото с включенной Smart HDR. На увеличенных фрагментах снимка нетрудно увидеть, что с помощью Deep Fusion намного лучше передана детализация бороды (видны даже волоски), структура кожи и узор рубашки.
Стейльман заметил, что вырос и размер файла. Если снятый на iPhone XR снимок Smart HDR занимает 1,4 МБ в формате HEIC, то на iPhone 11 аналогичный кадр, обработанный Deep Fusion, займет уже 2,6 МБ.
Какие iPhone поддерживают Deep Fusion?
Поддержка технологии Deep Fusion предусмотрена только в iPhone 11, 11 Pro и 11 Pro Max и более новых моделях смартфонов Apple. Это легко объяснить – обработкой снимкой занимается новый 8-ядерный нейронный движок, установленный в процессор A13. Он присутствует только в новых моделях iPhone.
What Is The Deep Fusion Camera On The Iphone 11
Smartphones aren’t complete replacements to professional cameras just yet, but Apple makes the iPhone a better camera every year.
This feature is available on the iPhone 11, iPhone 11 Pro, and iPhone 11 Pro Max. These phones were released with Apple’s iOS 13. They came with several significant improvements to their camera setup, including improved sensors, an ultrawide-angle lens, a night mode, and slow-motion selfies. However, one improvement that didn’t come out of the box with their newest flagships is the Deep Fusion Camera, released with the iOS 13.2 update on October 28, 2019.
Apple’s Phil Schiller described it as “computational photography mad science.” While many smartphones are making great strides towards improving image quality in very dark environments with Night Mode and very bright environments with HDR, most of the photos we take fall somewhere in between. The Deep Fusion Camera is supposed to reduce noise and significantly improve detail for photos taken in medium to low-light conditions, mainly indoor shots.
To demonstrate, Apple used several samples of people wearing sweaters—an item of clothing that frequently loses detail in photos. The sweaters and other items in the shots taken with the Deep Fusion Camera are more detailed and retain their natural texture.
RELATED: The Best New Features in iOS 13, Available Now
How Does It Work?#
According to Apple, the new mode uses the iPhone 11’s new A13 Bionic chip to do “pixel-by-pixel processing of photos, optimizing for texture, details, and noise in every part of the photo.” In essence, it works similarly to the iPhone camera’s Smart HDR, which takes several shots at varying exposures and combines them to maximize the clarity in the finished image. Where they differ is in the amount of information that needs to be processed.
What Deep Fusion is doing in the background is quite complicated. When the user presses the shutter button in medium light, the camera immediately takes nine pictures: four short images, four secondary images, and one long exposure photo. It fuses the long-exposure with the best among the short images. Then, the processor goes pixel by pixel and selects the best elements from both to create the most detailed photo possible. All of that takes place in one second.
When you initially snap a photo, it immediately starts post-processing the image in your album. So by the time you open your camera roll to take a look at it, it will already have the effect implemented. This is made possible by the A13 Bionic chip, which is the strongest processor ever put into a commercial smartphone.
How to Get Deep Fusion on Your iPhone#
You need an iPhone 11, iPhone 11 Pro, or iPhone 11 Pro Max to use Deep Fusion. It will likely be compatible with future iPhones, too, but it’s not compatible with the hardware in past iPhones.
Your iPhone needs iOS 13.2, too. If it’s running an older version of iOS, Deep Fusion won’t be available. To update your phone, go to Settings > General > Software Update. Make sure you turn on your Wi-Fi.
When your phone is updated, go to Settings > Camera and turn off “Photos Capture Outside the Frame.” While it’s a handy feature, it’s incompatible with the Deep Fusion mode.
How to Use the Deep Fusion Camera#
One of the other features Apple introduced this year is Night Mode, which uses multiple photos to come up with the brightest image. It is accessible via a toggle in the camera software or is automatically activated in very dark lighting conditions.
Unlike Night Mode, there’s no way for the user to activate the Deep Fusion mode manually. There’s no indicator that it’s even turned on. Apple’s AI automatically detects when an image is best suited for Deep Fusion and takes the shot in a way that is invisible to the end-user. The reasoning behind this is that they want you to be able to take the best photos during normal lighting conditions without worrying about which mode to pick.
However, there are several conditions where you can’t use the Deep Fusion Camera. At the moment, it’s only compatible with the wide lens and telephoto lens. Photos that are taken with the ultrawide camera default to Smart HDR if the lighting conditions are sufficient.
Also, since each image takes a second to process, it is not compatible with burst photography.
How Do You Know It’s Working?#
In many cases, you can’t know if Deep Fusion is doing anything. On paper, the technology is a huge leap forward in mobile photography. In practice, the differences may be a bit difficult to notice for most people. This is especially true if you’re not comparing two images side by side.
Deep Fusion is mostly noticeable in objects with a lot of textures. Things like hair, detailed fabrics, textured walls, fur, and some food items will be more detailed, especially if you zoom into them. With the holidays coming up, expect to see some very detailed images of people donning sweaters in your feed.
Is It Coming to My Phone?#
Deep Fusion will only be compatible with the iPhone 11, 11 Pro, and 11 Pro Max. Apple’s older devices, including the X and the XS, do not have the A13 Bionic chip, which powers much of the new camera processing features on the latest models. It can’t be added in a future software update.
On the other hand, if you’re not on an iPhone, it’s definitely not coming to your phone. However, other smartphone manufacturers, such as Google’s Pixel line, will likely see this as a challenge and develop their processing tools to rival Apple’s new mode.
Как работает технология сверхчетких фото Deep Fusion в iPhone 11. Примеры
В закладки
Пользователи уже активно тестируют технологию Deep Fusion для автоматического улучшения фотографий, появившуюся в iOS 13.2 beta 1. Она комбинирует лучшие моменты с нескольких разных снимков в один, обрабатывая их нейросетью и выдавая лучший, наиболее детализированный результат.
В обычном режиме фотографирования вы не заметите, что функция включена. Она работает автономно.
Для работы Deep Fusion необходимо отключить опцию Захват фото вне кадра в меню Настройки -> Основные -> Камера.
Важно: снимки, полученные через эту технологию, прогружаются спустя 2-3 секунды. Если этого не произошло, просто приблизьте фото.
Если вам не нравится результат, добавьте любой эффект в редакторе приложения Фото. Изображение станет мягче.
Примеры фото
Пример 1. Обратите внимание на надписи на линейке, с включённой технологией они получились очень чёткими, а с выключенной — размытыми.
Слева: Deep Fusion включён, справа: Deep Fusion выключен
Пример 2. Посмотрите на тумбу повнимательнее. На правой фотографии видно, как единственный на ней волосок хорошо выделяется, а не смешивается со всем фоном. Зелёная отделка стала более детализированной с iOS 13.2.
Слева: Deep Fusion включён, справа: Deep Fusion выключен
А вот снимки, сделанные Себастьяном Де Виттом. На ободе динамика с камеры iPhone 11 с iOS 13.2 видны более четкие текстуры, чем на iPhone 11 Pro с iOS 13.1.2.
Или другое фото с объективом при слабой освещённости. Видно, что надписи на объекте получились чётче, а шумов стало немного меньше.
Снимки сделаны на сверхширокоугольную камеру.
Слева: Deep Fusion включён, справа: Deep Fusion выключен
Слева: Deep Fusion включён, справа: Deep Fusion выключен
Пример от Тайлера Стейльмана. Deep Fusion отлично захватил бороду, лицо и некоторые элементы свитера.
Слева: Deep Fusion выключен, справа: Deep Fusion включен
Слева: Deep Fusion выключен, справа: Deep Fusion включен
Слева: Deep Fusion выключен, справа: Deep Fusion включен
Ниже очередной наглядный снимок от блогера с канала iPhonedo. Обратите внимание на линии.
Слева: Deep Fusion включен, справа: Deep Fusion выключен
А вот блогер с канала Mark’s Tech решил сравнить не фотографии с двух айфонов, а с Samsung Galaxy Note10+ и iPhone 11 с iOS 13.2. И знаете, «самсунг» сделал настоящую кашу, тогда как у айфона получился достаточно чёткий снимок.
Слева: iPhone 11 с Deep Fusion, справа: Samsung Galaxy Note10+
И напоследок. Фото листа кленового дерева от пользователя iMZ. На фотографии с Deep Fusion видны высокая детализация и текстура, тогда как без технологии перед нами размытый лист.
Слева: Deep Fusion включен, справа: Deep Fusion выключен
Что в итоге
Сейчас различия не всегда легко заметить, но уже видно, что технология работает. Фотографии, сделанные с помощью Deep Fusion, получаются более детализированными на обычной, сверхширокоугольной и телефото камерах. При приближении теряется меньше элементов, а шумы на снимках сглаживаются.
Пользователи отмечают, что полученные фотографии с Deep Fusion весят примерно в 2 раза больше, чем во время обычной съёмки. Тот же Стейльман говорит, что фото с iPhone XR занимает 1,4 МБ места, с iPhone 11 — 2,6 МБ.
Это только первая бета-версия iOS 13.2. Со временем Apple может улучшить работу технологии.
В закладки
How iPhone 11 Deep Fusion Works — AI Computational Photography
Apple Deep Fusion is a major feature in a camera-phone designed for the consumption of the masses. Similar techniques have been used in the past to get better results from the high end cameras [1] to stay relevant with the advent of better camera functions in people’s smartphones, which are able to rival them with their relative ease of use as well as the quality. Now the scenario is possibly going to change, with the onset of Apple’s Deep Fusion. In the words of Apple’s Phil Schiller “The computational photography is a Mad Science”[5].
How it works
Apple has included three cameras [2](visual in the image below) in their latest device that will, each one generating multiple images with variable aperture controls in order to maximize the capturing of the required attributes. What it effectively comes on to is that when you click your device to capture a frame in the specific mode, your camera will actually capture four different images or frames, one of these frames or images will be a long exposure and the subsequent ones will be four secondary images. These different frames then are combined to get an image output that has all the attributes of the captured scene.
In total there are nine different instances that are fused together to get the best possible image, the fusion isn’t compulsory, the system is smart enough to determine if one of the images that have been received from the cameras with variable shutter speeds are actually good enough or not to be considered for fusion or have to be selected independently of all the solutions. The process of capturing of the image as well as the processing of the neural engine is actually completed within a second [3], which is quite a remarkable [6]step given the resolution of the images and the known computational lag the deep networks exhibit even in the inference mode.
The Power — Under the Hood
The ability to generate inference on such high resolutions at such speeds is indeed a very remarkable step and a huge achievement by many yardsticks. The real magic however is not in the AI alone itself, the effective use of technology is also attributable to the cutting edge technology of the power boosted A13 Bionic chip [4] which makes all these supernatural speeds of calculations possible in the real world situation and gives definitive results in real-time. The domain of computation photography has been existing since a significant amount of time, but it was never possible to use it in real-time for practical implementations as the speeds were slow in the best case scenario. The introduction of new chip gives Apple the edge required to allow the use of the technology for real-time implementations.
Comparison with the Next Best
Previous year Google[7], one of the big names[8] in the field of computational photography, introduced Night Sight [9], which is a marvelous combination of camera hardware and intelligent image processing, which is capable of transformation of images taken in the darkness into having great detail for color and contents. Apple was expected to bring a similar product that was able to rival the features that are provided by the Google Night Stand and the response from apple is the introduction of Night Mode [10] on the new iPhone 11. The Night mode is capable of a number of mind-blowing features, whereby the images, taken in complete darkness that would in the previous versions will have absolutely no features visible can now have great amount of details captured in itself, as is evident in the comparison of images from the previous version of the iPhone itself to the latest light mode.
How AI Helps you to Become a Better Photographer
An image has a great many details captured in itself, the latest technology has enabled the automation of the tasks that otherwise would have required a large amount of time and dedication in the learning procedure. One such example of the same can be the auto-focus[11] features that is nowadays so common in almost all the devices was up until a few years took a great deal of effort to learn that is now so effortlessly executed almost daily without a single day of training. The vision in the introduction of the discussed details actually attempts to repeat the success of the same, only instead of the hard coded logic we have artificial neurons which have learnt important characteristics on a wide variety of images and are now capable of making decisions on the important features as well as their intensities in a given image and can now extract important information from a given set of images (taken from the different cameras in the new iPhone with variable features) and integrate them into a single representation of the frame with all the information intact and the resultant image that can actually rival a professional.
Learn More
If you’d like to learn how to develop your own AI apps in Computer Vision (AI-CV) check out our training called Ultimate AI-CV Practitioners PRO. This course will teach you how to become a PRO AI-CV developer with: Object Detection, Instance Segmentation, Pose Estimation, Android and Raspberry Pi Development! Click the link or image below to attend the FREE Webinar. — Click Here