Smartphone photography has changed the way people look at imaging and optics. Today, you don’t need to carry a bulky camera with a variety of lenses every time you travel. If you have a good smartphone camera, it will take care of things pretty well. Now, to make smartphone cameras even better, OEMs are doing a lot behind closed doors. It doesn’t just involve partnering with older camera brands and putting bigger camera sensors on smartphones. It requires complex R&D, lots of technical challenges, and a lot of trial and error.
To understand what is going on in camera development, we had a small chat about Vivo’s camera development during the pre-launch briefing of Vivo X90 series with Keshav Chugh, Product Manager, Camera Development, Vivo Did. Having a background in photography, Chugh takes care of Vivo’s camera development, including what new features to come up with, what hardware to use, how to tune the software according to different markets, and more. Here’s how the conversation went:
1. To begin with, let us have a proper understanding of what is behind the camera development in Vivo smartphones?
So every time we start out with a new device, there is a three-step system that is usually followed to tune up the cameras. The first step is coming up with a problem statement. A problem statement is what we want to achieve with the camera, what will be our focus for a given product.
Next comes the optics part, where we choose which hardware to use for which device. The optics step is also where Zeiss steps in. They are experts in optics and imaging, and they help us tune the hardware for different markets.
The third stage is computation, where machine learning and AI play a big role. This is where after everything is in place, we look at the software and how we can use our algorithms to enhance the camera experience. This includes not only the camera app or controls, but also how AI enhances the images captured by the camera. How much light to let in, how to tune the colors, edge detection in portraits, all post processing is decided and tuned in this step.
2. Can you tell us more about the Vivo V2 chip and what is the company’s aim with the ISP chip?
Yes, so apart from handling all the camera processing that Vivo (and iQOO) flagships do, the V2 chip also acts as a Deep Learning Accelerator (DLA) which facilitates and handles all the AI algorithms. V2 is a scheduling chip that organizes, controls and optimizes workloads in the production process. It basically acts like a manager to help solve problems and implement new camera features.
Coming to development, we first need to figure out what problem we are trying to solve. Then, we’ll learn how to develop an ISP chip. Then, our development teams kick into action, third parties who help us also come into play at this stage. Ultimately our goal is not to market the V2 or anything for Vivo, the ultimate goal is to solve a problem. For example, if I’m telling you that the Vivo X90 Pro has the best night camera in the market, I should have results to back it up.
Plus, with the amount of AI involved in creating features like the Zeiss Cine Flair or our Portrait Mode among other things, we need hardware support to run so many algorithms.
3. Tell us more about Vivo’s partnership with Zeiss
With Zeiss, we have two types of cooperation — one is co-engineering, one is collaboration. The co-engineering part is where they work with us product-by-product. For example, how they helped us with camera development on the Vivo X90 series and other previous X-series flagships. That’s product engineering, where they’re with us every step of the way in the development of the camera.
Next is collaboration, which is a more long-term thing. Here, we are currently doing R&D on what we will bring in the future. There is a lot of experimentation and testing involved. Zeiss in Germany has a specialized laboratory exclusively for the development of VIVO products. That’s why here also we work on problem statement basis. For example, how do we bring Zeiss Cine Flare to smartphones is a problem statement. Now, all it takes for the user is pointing and shooting to bring that cine flair to their images. But it is very difficult to implement, as we have to replicate that natural cine flair using AI.
This includes figuring out what the scene is, where the light is coming from, what the subject is, what the light source is, what the white balance is in the scene, how intense the light is, and things like that. Based on that, the camera has to calculate around the scene, the scene is simulated, then there are different types of cine flares. There too, we have to calculate what size flare I need based on the lighting at what location I need it, the color of the flare based on the lighting scenario and white balance. So there are a lot of AI applications involved in all of this. It’s not just slapping on a filter with a PNG sticker on it, just for the sake of it.
This is another reason why our partnership with Zeiss has been successful. Because if we just slapped on filters, a big legacy brand like Zeiss wouldn’t want to be associated with us. It is because they are happy with the final product that we can use their name with confidence.
4. Who is building the process for V2? Does Vivo have any plans to license these ISP chips?
So the silicon fabrication is done by the silicon manufacturers, but the design and architecture is proprietary to us.
Talking about licensing, there are no plans to license it. That’s not the idea behind the Vivo V-series chips. It’s something we’ve invested a lot of money and resources into and it’s our IP.
5. How has Zeiss responded to the positive response Vivo’s new smartphone has been receiving for cameras?
That’s why Zeiss is with us every step of the way, even when we’re tuning the sensor. That’s why we’re calling it a Zeiss sensor, even though it’s a Sony sensor, hardware wise. But in the optimization and tuning of the sensor, Zeiss plays a big role. Zeiss is on our side when it comes to colors too. We are one of the few OEMs doing multiple color science. We have Vivo Vivid Color and Zeiss Natural Color. With the Vivo X90 series, Zeiss is also helping us calibrate the display professionally. Because if our screen is not calibrated, then there will be no point in taking images in Zeiss Natural Color.
So the partnership has evolved beyond cameras and towards the development of the phone as a whole. Also, the Zeiss Natural Color display setting is not just a software that Zeiss is pushing, we are calibrating the display individually.
6. What was the biggest challenge while developing the cameras for the Vivo X90 series?
I would say the most difficult thing would be tuning for the 1-inch type sensor. Now, we can do very simple tuning or we can do very complex tuning. Very simple tuning is done by some brands and very complex tuning when you localize the phone. Every country has different demand. With India being the most important market for us, tuning our cameras according to Indian customers, be it in terms of skin tone, portrait, or front or back cameras, or understanding their needs, how they differ from other users, is very important to us.
Now, getting all this right and turning it into a real product that not only works, but is the best at what it does, is the hardest part. It also takes a lot of time. For example, I would have to run hundreds and thousands of simulations to find the right parameters. Then, we put it in a trial version, where we do a real test. Then we repeat this process until we are happy with the end result. Only then do we feel that a product is ready to go to the markets.
7. What is your favorite camera feature in Vivo X90 series?
In terms of camera, I would say that I love the portrait modes. Admittedly, the Cine Flare is a new one, but overall all of Zeiss’s stills are my favorite. So before that I was a commercial photographer and I love street photography. In street photography, 50mm is the perfect focal length. Pair this with the f/1.6 aperture, and it becomes an incredible thing for street photography. As such the (f/1.6) aperture is huge. It lets in a lot more light and this time we have a 30% bigger sensor, so portraits have improved a lot compared to the previous generation.
8. One camera feature you want on the next Vivo flagship?
Now there isn’t a single feature that I can think of. As product managers, we want to have as many features as possible in our products. Although, just one thing to think about, I would love to have a 35mm lens on my smartphone, but it just isn’t possible. While I am a product manager, I am also a photographer so my demands are the same.
This is not possible because even though we are providing services to professionals also, there is a balance to be maintained. That balance is you have to have a great camera while still having a great smartphone. It cannot be a good smartphone with a bad camera or a good camera setup on a bad smartphone.
9. So you must have seen Xiaomi’s concept phone with replaceable lens. Is something similar coming from Vivo or is Vivo working on replaceable lenses in smartphones?
No. Look, all brands have their own ideas and they never hit the market. The idea behind these is that OEMs are constantly trying to figure out what the future of the smartphone industry will be.
Now, this is a great thing for the industry because it opens up new ideas. And in the future, concepts like this are important if we want to figure out how smartphones can completely replace traditional cameras. Even we have a lot of concept phones – some we bring out, some we don’t. Still, these kinds of concepts are a good thing, not just for brands, but for the industry as a whole.
10. Do you think replaceable lenses on smartphones are a good idea?
As a photographer, yes. But as a smartphone user, no. Because the technology doesn’t exist yet. While the idea is good, the technology is still not up to the mark. My personal opinion is that if you have a 5-inch thick lens on your smartphone, you can only take one camera and have a bigger sensor. Second, if my phone always needs a lens attached to the camera, that means my camera is useless without a lens. As I said, it should be a good smartphone. So, if I get a smartphone camera that allows me to attach different lenses, I’m going to have to carry multiple lenses at least at some point. Now, if I’m carrying that much stuff, why not get a proper camera?
