Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Yes, Larry Page’s first crawler was in Java, and Scott Hassan rewrote it in Python.

From https://www.vanityfair.com/news/2018/07/valley-of-genius-exc...:

> Scott Hassan: In the fall of ’95, for some reason, I started hanging out with Larry in his office. . . . At the time, Larry was trying to download a hundred pages simultaneously. And I was fixing some of the bugs that he was having with Java itself, and this went on for weeks, if not months. And I remember thinking, Wow, this is insane!, because I was spending a lot of time fixing this underlying tool. And so one weekend, I just took all his code, I took his whole entire thing, and threw it all out, and rewrote the thing that he’s been working on for months very quickly—over a weekend—because I was just sick and tired of it. I knew I could get the thing working if I used a language I knew very well, called Python. I wrote it in such a way that it could download 32,000 pages simultaneously. So Larry went from barely downloading a 100, to doing 32,000 [pages] simultaneously on a single machine.



32,000 pages simultaneously - in 1995? Async has only recently been added to Python, no?



Doesn't select have a maximum of 1024 file descriptors it can handle at any one time? Or some such?


By default, yes. But it can be increased. See limit(1) and ulimit(1).


is that a fact or are you guessing?


select doesnt scale well.


Using threads maybe? Id be curious to see how you would achieve that in Python easily back in 1995.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: