Note this is not aerial imagery shot from an aircraft with an Apple phone, this is someone with an Apple phone on their head cycling around the neighborhood and then viewing the 3d model from above.
No less cool, but I was rather confused for a lot of the article, wondering what in the world the range on this sensor is (usually you get a few meters on normal-priced sensors, not hundreds of meters, and the picture showed entire streets being captured). The article actually mentions the maximum range is 5 metres after you set it to low confidence.
I would suggest "birds-eye" or something similar. That's sufficiently ambiguous that the source doesn't matter - e.g. an ortho is essentially a top-down view of a 3D-modelled surface. I guess most people outside GIS/Remote Sensing are unfamiliar with the terminology though.
The problem (I guess) is that aerial/satellite/orthorectified are not equivalent.
> The problem (I guess) is that aerial/satellite/orthorectified are not equivalent.
Yep I agree - I would call it orthorectified georeferenced imagery but unless you're in the GIS/OSM space you probably don't know what that means. I agree that "birds-eye" would have been a good alternative.
Satellite imagery is technically incorrect whether it's from a iPhone or a plane but that's what most people call it and know it by. (I love this thread on how "satellite" almost became known as "bird mode": https://twitter.com/btaylor/status/1099370126678253569)
I chose aerial imagery as an in-between - understandable to everyone, still not correct but less incorrect than satellite imagery.
Not an issue of yours, it's all a matter of context -- and how closely one reads the article while trying to get an idea of what it's about. I just happen to be in the OSM ecosystem deep enough to be very familiar with the word aerial (as opposed to other types of area photographies), but not deep enough into the Apple sphere to know the capabilities of these devices and realize what the title clearly must mean. It might just be a me-issue :)
Do you know the functional limit of the iPhone’s LiDAR? Ie: could I strap my Pro to my old DJI drone and create some photogrammetry of my house or whatever?
> could I strap my Pro to my old DJI drone and create some photogrammetry of my house or whatever?
Yep that should be possible - as long as you flow below 5 metres though!
If you have a drone and are somewhere that is legal & safe to fly, photogrammetry (3d model from just visual photos) will likely give a better result than relying on the iPhone LiDAR stitching.
Set the drone image self timer to 2 seconds, fly around for a bit and then upload all the photos into WebODM: https://opendronemap.org/webodm/
You can run it on your local machine (see instructions in blog post) or hosted for a fee. It creates brilliant models and is a lot of fun!
I've never gotten photogrammetry from a drone to work properly. Frankly the headache is just not work it in my opinion, especially since it usually entails flying low enough to annoy your neighbors.
That's not my experience at all - even just driving the drone in a straight line with a 2 second (repating) self timer on I get great models processing in WebODM with the default settings.
> it usually entails flying low enough to annoy your neighbors
In my experience the higher the better - you can get a really nice model from 120m (legal limit in Australia).
I wrote this guide on creating orthorectified imagery using just an iPhone, using a demo on George St, Sydney!
Usually this is easier with drones, but there are a lot of places you can't fly drones. I hope this guide comes in helpful, especially if you're contributing to OpenStreetMap.
I've found you can strap an iPhone to the front of a bicycle and if you go slowly the 3D models/LiDAR point clouds are quite good (it has a 5m range). There are drift issues if you try to complete loops though.
Have you tried Scaniverse for the capture step? It does a decent job at correcting for drift and avoiding misalignments for these sorts of scans. For example:
Ever since iPhone introduced Lidar, I've wondered why Teslas got rid of it. They said it was due to additional cost and complexity in models. Of course Waymo heavily uses Lidar but they have huge sensors they mount on cars to do so. Why is that lidar so different? What's stopping Tesla from adding the same sensors the iphone has, but all over the car and stitching it together like it does video from cameras? Is it just to do with distance? Refresh rate? Would love someone who knows this to teach me!
There are a lot of different things called "Lidar" : at a minimum, Structured Light, Time of Flight, and triangulation are the three most common methods. Within each method, there are vastly different approaches - scanning vs. single-frame, frequency modulated vs. pulsed, and more.
Each approach comes with different trade-offs suited to different use-cases. In the case of the iPhone rear camera, I believe the approach used is single-frame pulsed Lidar using a VCSEL (vertical cavity surface-emitting laser), a dot-grid diffuser, and SPAD (single photon avalanche diode) CMOS camera.
Basically, a shifting grid of dots at a very specific wavelength is projected outwards from the laser, and an unbelievably, photon-level sensitive camera tuned to that precise wavelength measures how long each dot takes to come back after it's emitted. In this way, Time of Flight can be measured and a precise distance determined.
This system provides incredibly high resolution at a moderate distance (in this case, 5m or so) at a moderate cost.
For a car, this system wouldn't be very plausible due to range, resolution, and output power constraints. The size of laser required to emit a coherent dot matrix which could reflect at a long distance would grow to an extreme degree, the noise floor of the SPAD would eventually render the system useless, and the amount of data required to produce a higher resolution dot cloud would become quite challenging, as these systems generally need to be heavily filtered and sub-sampled, so an enormous amount of input data processing is required to produce a reasonable output.
The main reason that Tesla don't use Lidar is that they decided to bet on optical (IMO, foolish), but the reason they decided to bet on optical is complex and definitely in no small way cost driven. They couldn't just slap an array of iPhone sensors (I believe they're Sony-manufactured) on a Tesla and call it good.
Tesla did not add iPhone style lidar (up to 5m) because they already had 12 cheap ultrasonic sensors (up to 8m) for short range distance detection since all they need to know at that range is will or won't I hit something. They are now full optical though and may have already been ignoring some sensors in software. https://en.wikipedia.org/wiki/Tesla_Autopilot#Hardware
There are several reasons why they would not just add new sensors even if they are in some/all ways superior:
* Tesla special cases different driving situations. This makes sense as a product company seeking to sell consumers on solutions to each situation as a feature but means they need to re-implement each situation individually when the sensors change rather than just swapping out some generic distance detector net. You can observe this from the loss of features when Tesla went full Vision over the last two years.
* Adding new sensors adds new errors which leaves you to chose between ignoring danger conditions present in only one sensor as possible errors or greatly increasing safety stops etc (some of which would be genuine previously undetected dangers). Because of Tesla's engineering culture the prudent second option or a sensible middle ground would be impossible to make since they would fail from a user experience point of view at the first false stop during a demo to the big man and then be forever poisoned (imho).
* Tesla has always been pursuing a full vision only solution on the reasonable assumption that if humans can do it then eventually computers will too. The only disagreement with that strategy is when exactly that will/should be.
I like how you ignored the truest reason: they don't want to pay for it, and they don't really care if the current hardware can actually solve FSD.
There's no way you can argue it was done in good faith when it was an arbitrary transition where some functionality hadn't even been solved yet and was disabled if your car didn't come with ultrasonic sensors. And to add to the comical levels of disregard for buyers, the official advice for finding out if your order would have ultrasonic sensors was: "Look for the 12 sensors on your vehicle", ie you didn't know until you paid and it was a roulette wheel.
I don't think the price of a $10 sensor would trump Elon's desire to be right about vision. Price was a factor for not using the larger, longer range lidars though since that cost more than planned price of the vehicle ~10 years ago.
Now it's not like there are no ultrasonic sensors to be had, but they would have had to pay for tooling changes, design verification, assembly changes... easily 10s of millions of dollars for what you're brushing off as "the price of a $10 sensor"
Their options were remove the sensors and be transparent about why, or throw up a smoke screen with idle gibberish about vision was definitely better despite needing to cut features that had been practically solved nearly a decade earlier. Guess which they chose.
> Price was a factor for not using the larger, longer range lidars though since that cost more than planned price of the vehicle ~10 years ago.
Good thing I didn't say or remotely imply that?
Overall I think your type of blustering works better on people not working in the AV field.
> Adding new sensors adds new errors which leaves you to chose between ignoring danger conditions present in only one sensor as possible errors or greatly increasing safety stops etc
No, sorry that cannot be left as is.. I see your arguments but they are pulled of thin air, or is there any source that reasoning actually happened there. If fusion done right, this will never increase but decrease errors.
> they would fail from a user experience point of view at the first false stop during a demo to the big man and then be forever poisoned (imho).
And failing on the first situation that a second sensor set would have clearly prevented can easily be argued the same.. this is a strawman.
Lidar I believe can be heavily affected by weather and might also require a lot of expensive sensors. It works great for VFX work when you need a precise model of an area. But I would guess the refresh rate is also not amazing.
I live near Waymo and see the cars all the time - they do have huge things on top but the weather in Santa Monica is also very good. Not sure if I have seen them ever driving in the rain?
Interesting. I’ve used a Velodyne VLP-32 in reasonably heavy rain with reasonable performance after some error correction. I was on foot and not driving, and probably only for a few hours total. But it seems to me that a big company with serious engineering talent should be able to make it work.
Yes.. any other car manufacturer I know of (likely not complete) is considering also Lidar sensors... they are still pricey and huge, but there are also MEMS-like improvements (likely incorrect terminology) and also costs get continously lower.
At the same time advancements also happen at the multi-model side, not fusing different deep learning outputs afterwards but right in the one model (not judging if that is a good idea... but the steps taking currently seem to get bigger and better).
Lastly, also camera has degraded performance under certain weather conditions.. some even say our eyes can be prone to that! :D
So that's just all the challenges that need to be taken. Yes, humans can drive with their eyes only, but is that perfect? Sometimes we e.g. would also benefit greatly from radar/lidar, as e.g. in dark conditions our judgement of distances is really flawed.
Refresh rate of some of this already aged car lidars was like 10 Hz, good enough I'd say? And it is truly amazing what details one can get out of those when getting close and move around in those point clouds.
It's been a bit since I've been out of touch with the autonomous driving space. Was there significant progress the last 3 years? What are the current challenges?
Simply because they are all in on cameras although there are recent teardowns of their cars online that foreshadow a possible comeback for additional sensors.
This is getting very close to actual field survey work and IMO the best app, by a mile, to carry out the scanning part in this kind of setting is Dot3D. It's basically been built for larger-scale capture and reliable loop-closure.
Here's a scan of several km (!) of length all done with Dot3D on an iPad [1]. You can also explore the point cloud on the web.
Also, digging a little deeper, here's a study comparing the accuracy and loop-closure capabilities of several 3D scanning apps. It's spot on. [2]
It's a blog with only one post. Though it claims to be a comparison of tools, it only mentions the name of Dot3D, refers to all other tools by number, so you can't look at them, and is written by someone on the Dot3D team.
300$/y is a bit much for hobbyist street scanning though... The linked content marketing blog post shows Scaniverse to be nearly as good, so that's nice to know as an alternative.
I recently 3D scanned an area of my property (about 250 feet long) with my iPhone in order to make a map. Polycam was the winning app (Scaniverse and SiteScape did not work as well).
Do not underestimate the sheer amount of walking around and aiming your phone that this technique requires. A 5m range is not much, and in some cases the effective range seems to be less than that. If you just want a to-scale image of an area, you can stop after "Capturing the model" and take a screenshot of your captured scan, but it's still a lot of work.
Nothing against the iPhone, but it would be neat to also have (e.g. open source or DIY electronics) projects which enabled use of lidar tech at home-user prices.
It's worth noting (again, without trying to be a grinch - merely trying to add to the overall picture) that there are actual dedicated lidar scanners on the retail market, of higher spec and capability, within a similar price range as the Apple devices involved here.
Of course, you don't get an awesome iPhone with those! :)
> it would be neat to also have (e.g. open source or DIY electronics) projects which enabled use of lidar tech at home-user prices.
I agree! I've heard of "cheap" lidar scanners (Intel Lidar Camera L515 for $375 apparently [1], or [2]) but the act of turning real time depth measurements from a UART output into a 3D model (or even just a coloured pointcloud) is no simple task!
Something that may scratch the itch though - I've thought it would be cool to build a bicycle mounted, microcontroller-controlled servo mount for an iPhone which moves it back and forth to increase the the width of scanned area (I assume a sine wave).
Thanks for that - I've heard that feedback once before. I've changed the h2 to an h3. Do you have any tips for how I could make it feel less like the end of the page?
This is very cool. Would it theoretically be possible to "open source" a Google Streetview type of thing by getting "mappers" to contribute their walks around the city, with some app taking photos / lidar? In a similar way that distributed projects like SETI split the work among lots of machines (that maybe not the best analogy~~ha!)?
Similar alternatives exist - see Mapillary [1] (owned by Meta now, but they support OpenStreetMap and allow usage for that [2]) or KartaView [3] (owned by Grab).
I'm not aware of an OSM/open place to store/contribute pointclouds though - please let me know if you come across one!
No less cool, but I was rather confused for a lot of the article, wondering what in the world the range on this sensor is (usually you get a few meters on normal-priced sensors, not hundreds of meters, and the picture showed entire streets being captured). The article actually mentions the maximum range is 5 metres after you set it to low confidence.