These topics are extremely hard to solve from scratch yet they are distilled pretty well in the above articles and they include a further reading section.
I would implore others to have a gander. I wish the same could be said for their documentation.
Using the Prime video app on my Android TV it seems like some basics aren't really attended to.
The aspect ratio on the thumbnail for film I watched was stretched the other day, which is not the lack of attention to detail I'd expect from such a rich company.
Apart from this, it just isn't a smooth experience to navigate or search in. As usual in development nobody is paying any attention to responsiveness.
There is no reason this app shouldn't work as well as the youtube app in these respects.
Amazon products are never polished or fully fleshed out. You can spend a few minutes on their flagship site and find weird design choices or basic CSS mistakes.
A shame considering the amount of money they have. You'd expect a bit more, but then again Amazon are in the business of scale, quantity over quality.
At least it keeps them from having a monopoly on literally everything.
The whole point is to use multiple Availability Zones across Regions, to protect your uptime from failures in single AZs or entire Regions. Amazon.com does that.
Note that pager duty now weathers Amazon outages beautifully, but the point is that amazon doesn’t have 100% uptime and that properly designing distributed systems is very hard.
This is something Netflix has been doing for a while. I think it's called adaptive bitrate streaming - the video quality automatically adapts to your network connection.
A frustrating work around. But a good router like from ubiquiti should have qos options to throttle your streams or a single computer on the network from hogging the bandwidth.
It truly baffles me how one could devise an interface so bad as the PrimeVideo UI. I have yet to discover the algorithm behind the near random search results trows up, and autocomplete for 90% of the cases trows up something 'that is not available in your region'. It's 2020. How hard can it be to serve a small catalogue of available titles and allow some decent parameterized querying?
I also always wonder how companies can simultaneously offer to sell their highly advanced AI-bla search, yet utterly fail at searching themselves.
I still remember the time when Google tried to license their search servers to enterprises. It appears that market has now been completely eaten by Algolia. And my hunch is that it's because Google's search results are completely irrelevant for professional users. I search for a Windows API function name, and I get pages of SEO spam trying to sell me an unrelated Udemy beginner's course.
It's truly amazing. You need to type way too much of the title, and even then you may not get that, you may get seasons 1-45, which are inexplicably different items on the result, of something you didn't search for at all.
Of the 3 services I subscribe to, only Amazon gets things so wrong so often.
Funny story, just yesterday some friends and I were trying to watch "You're Next" as it is free with Prime. We found the properly spelled entry but it only offered to add it to a watchlist. There was another entry, titled "You're Next'" (note the trailing single quote) that was the actual movie and available to watch. My only guess is some peculiar manual data-entry mistake but that doesn't really explain the correctly titled entry...
Do you know of any articles on what is wrong with Prime UI? I am writing up a proposal to fix a similar service that also has really bad UI - I would like to see if there are intersections between them.
No article I know of, but I'm happy to start a chain of complaints here:
- When you search for a title, even if that title is in their library, odds are the exact match won't be the first result
- The differentiation between what is rentable, purchasable and freely streamable (with gradations based on different subscriptions) is incomprehensible from the listings pages.
The following is just for desktop browsers, since I haven't tried their app:
- they split their shows by season to make their catalogue look bigger, but it's more confusing for the user
- don't make it explicit on the thumbnail either, so you have to hover over it to know what season of the show we're talking about
- although they do a decent job at continuing to watch episodes in a season, they're not making it clear when you've already watched entire seasons and those remain on the screen and are recommended to you over and over again
- advertise a trailer as an entire new season (see: The Boys season 2), leading to pretty gigantic disappointment (vs the netflix experience where a red notification for something you like is actually a great surprise because you KNOW it's an actual release)
- have a lower chance of being able to reload a page of something you left mid-watch and the page successfully reloading to the right content and timestamp
- the overflow of their preview hides the show underneath the current one which means any directly vertical scrolling is infuriating because you will trigger the overflow of the show underneath (that you see 20% of) only after having gone vertically down beyond 80% of the div in question
- the line = product line thing is confusing, as everything starts with "prime" for a full screen, (so why the repetition? oh, it's because ->) suddenly you get news (I don't care about them) and movies to rent or buy (I also don't care about paying prime to pay on top of it)
Don't forget that UHD versions are separate from their SD/HD versions and sometimes OV versions are separate from subbed versions that are sperate from dubbed versions. So you end up with ~6 search results for the same movie. Beware of what you buy.
Indeed. It’s also better than another UI which requires an esoteric feature of a remote control that I don’t even use and offers no way of switching that stupid choice off, preventing me from ever getting more information about the shows it is presenting beyond the title and banner.
It’s also better than one that always says I’m logged out when I start it, and then magically logs me in a minute or two later... while I’m in the process of trying to log in. (Or any of the several other ones that require me to log in nearly every time.)
Of course calling any of these a “User Interface” is almost laughable. SLOM is a better term. That’s “Small Library Obfuscatory Method” for the uninitiated. Or maybe they’re just trying to save bandwidth by making viewers spend viewing time searching pointlessly through their offerings that are ordered seemingly according to the pattern of pigeon droppings outside their office.
No, I didn't. Last I looked you couldn't at all. Apparently I can now, but not via my Roku. I have to figure out how to get to Netflix from a PC apparently.
Also note that you're usually limited to a certain video quality when watching on a PC. Unless thing have changed since I last checked, you can't go past 1080p on a computer; you have to be on a streaming device (Roku, AppleTV, etc.) This is why I usually watch streaming services through my Playstation even though I have the PC hooked up to my big TV.
Yeah, it's quite bad UX. The subtitles can also only be configured via the website, which is something not many people seem to know. On my TV the default subtitle settings result in blooming so I configured them to be yellow with a black border and transparent background.
I almost don't think any service will ever get a decent ui. It seems to be against the business model to be able to search easily and keep lists of what you have watched and what you want to watch.
But TBH I would just like to watch one video on prime without it being stretched out of shape. It is probably some weird edge case with my particular STB, but every other streaming service seems to manage it properly.
I am always astonished at how bad the UI of Amazon products are. I know that there is some dark pattern used on purpose, but even the "normal UI" is a disaster. It lacks clarity, it doesn't convey the information needed, it is often confusing ... How can the e-shopping leader can have a worse UI that a small business ? It keep boggling my mind.
I am seeing a new UI for the past few weeks, where the FF is 10s at a time, with frames displayed over the progress bar: much like Netflix. The newer experience is much better.
The UI on prime was so much better than the Hulu UI from about 2018-june 2020.
Someone clearly wanted to do something "different and more beautiful than Netflix". The result was basically scrolling through shows one at a time. search was bad. Horrible.
My wife and I tried to re-watch some episodes of Brooklyn 99, and every time we would watch them they would start 20s from the end of the show. Like they stored that we had last watched ending at the end (-0s) and now when coming back months or years later, we might want to resume watching the last 0s with a buffer before (a bad resume behavior).
It seems they finally dropped that UI I'm the last few months to something sane, maybe the same as before I don't remember.
You'd think that after 15 years or so of streaming video (probably more) they'd get making a UI for it right.
What I'm instead seeing is an attempt at copying Netflix' interface. This is actually far too common - competing services that just copy their competitor's homework when it comes to UI.
I mean it makes sense in a way if you're targeting that service's current members, they're familiar enough with the UI etc.
Sounds like cloud is getting ever closer to dedicated servers.
One would hope that for the 10x price increase over bare metal, someone would abstract away things like the underlying hardware or OS failing. But apparently, no, you have to pay the premium and do all the work.
The only use-cases that I could imagine for such a tool on EC2 would be if you either don't use containers, or if you oversubscribe your virtual servers by having higher container limits than what the instance can endure.
In the first case, the proper fix is to use containers. Docker can do CPU limiting for you so that one service spiking won't affect its neighbors on the same instance.
In the second case, I'd go bare metal and then hardware is so cheap that there's very little temptation to oversubscribe on RAM or CPU.
The core concern is not about the capabilities of the compute abstraction being used (bare metal, containers or functions) or testing OS capabilities. The aim is to validate mitigations which are in place to counter turbulent scenarios (For example: massive spike in traffic, network outage, dependency is down, etc). These scenarios generally originate outside the given system.
These kind of questions should be asked and systematically validated (quoting the article):
* Have you tested how the system behaves when the underlying instances have a sustained CPU spike?
* Is the system behavior understood under different stress?
* Is there sufficient monitoring?
* Have the alarms been validated?
* Are there any countermeasures implemented? For example, is auto-scaling set up, and does it behave as expected? Are timeouts and retries appropriate?
I believe we just have a rather different approach here.
"Have you tested how the system behaves when the underlying instances have a sustained CPU spike?"
Since dedicated boxes are cheap, I'd just buy 5x the CPU resources that I reasonably need and call it a day. If there ever is a more than 5x traffic spike, then docker will prevent it from being a noisy neighbor, so the affected services will just become slower than usual. But even a 10x traffic multiplier would just produce a 2x slowdown, which should be tolerable for most users.
I agree that on clouds you want to save costs by only booking what you need. But bare metal, you can usually afford to keep spare capacity around all the time.
As such, I wouldn't plan for the system to behave well under stress. I'd try to always have enough resources around so that stress never happens. At the end of the day, this seems like a developer time vs. resource costs trade-off and for most companies, developers are sparse and resources are plentiful, so they'll have a very different trade-off from big FAANG companies.
"For example, is auto-scaling set up, and does it behave as expected?"
If your system is usually 90% idle, I wonder if you'll ever need that auto-scaling. Also, I'd say my customers can endure it if page load time goes up from 100ms to 200ms. So in my opinion, there is little need for auto-scaling for most companies.
>"Have you tested how the system behaves when the underlying instances have a sustained CPU spike?"
You didn't really address this question, you addressed a different question, which is a traffic spike.
>Also, I'd say my customers can endure it if page load time goes up from 100ms to 200ms. So in my opinion, there is little need for auto-scaling for most companies.
100ms to 200ms average? What about the tail? Your app might go from P99 - 500ms to P95 - timeout. That's when you'll lose customers.
If the underlying hardware is a bare metal server, it won't magically turn slow and have a CPU spike. That problem is caused noisy neighbor and kind of exclusive to clouds.
Well, with the 2x example, my app might get from a 1s P99 to a 2s P99 which feels slow, but is still doable. Again, those timeouts are usually introduced by cloud infrastructure. For example, if you use nginx outside of Heroku, it won't have a 30s timeout for file downloads.
No, there is no different approach. You are misunderstanding what is being addressed in the article. This is not about bare metal vs cloud or autoscaling vs no-autoscaling/overscaling or developer time vs resource costs.
The article talks about injecting failures at various points in the system, understanding how the system behaves under this stress, putting in counter-measures for the resulting problems, and eventually re-running this to validate those counter-measures.
I did understand what the article is about. But you only need to worry about failure under stress if you are driving your system close to the hardware's limit.
In the virtual cloud world, that is common, because you rent the cheapest instance that will be big enough. In the bare metal world, that is rarely the case, because you usually get a Ryzen with 16+ cores and 128+ GB of RAM. In that case, there's no point in checking what will happen to your 200 MB web app if there's a CPU spike. It'll be just fine because the hardware can handle 10x the load without a hickup.
Similarly, if your page load time is dominated by internet latency, it doesn't matter if your CPU needs a few more ms to spit out the page HTML. So there, a 2x CPU usage increase will be barely noticeable to the user.
I fully agree. I'm usually working with companies in the $10mio to $100mio ARR range. So obviously my way of doing things will fail at Amazon's scale. But let's face it, most developers are not at FAANG but at normal mid sized companies.
Let's try to tie together what you're talking about (auto-scaling/capacity), with the OP and blog post was mainly about (chaos engineering/engineering-for-failure). Imagine:
- You operate a service with significant traffic, and through empirical experience, you have a good handle on what 1x traffic looks like, and have even seen spikes to 2x traffic on rare occasion, which your overall system handled just fine. Applying your overall philosophy, you setup your system to allow for 5x the CPU resources you need, and call it a day, nothing to see here.
- But, guess what? Unbeknownst to you, your system has some critical bottleneck that would only surface at 3x your usual traffic, which could be anything from hitting some misconfigured max limits on your load-balancer, or exhausting all your database connections, or running out of threads or inodes on your server hosts, or triggering a kind of retry-storm/brownout due to slowly increasing latency in one of your service calls that only explodes past a certain limit (due to some unintended interaction with your core timeout/retry logic), or any number of latent potential bottlenecks that you never knew about, because as long as your system stayed under the critical limit, it was completely invisible to you. In other words, these are non-linear failures, that you cannot simply solve by extrapolating out with "1x traffic = 1x # of servers, 5x traffic = 5x # of servers".
- As a result, not only do you don't have nearly as much head-room for scaling up as you think you do, but ALSO when you do encounter such a failure, you cannot easily just "scale out" horizontally, because the failure mode itself is only exacerbated by horizontal scaling. When you encounter such failures that break some axiomatic assumptions you have about your system, it can be incredibly difficult/painful to reconcile, especially if you had no plans and no knowledge about these invisible/latent aspects of your system ahead of time.
Chaos engineering isn't about scaling at all, not really. It's about finding latent defects in your system, by actively probing your assumptions and seeing if your system behaves as you would expect. Using traffic to generate stress on the system is just one way to introduce some "chaos", but there are many other ways too (as covered in the article).
Of course, it's also true that systems need to reach a certain minimum level of complexity, before the ROI of introducing chaos engineering becomes really worth it. You need to have a complex-enough set of services, dependencies, or interconnected components that are likely enough to behave in non-obvious ways, that you have to do independent chaos engineering to test them effectively, rather than simply reasoning about their properties directly.
My experience is that I have yet to work with a company where this level of failure-proofing makes financial sense. Purchasing more hardware than needed is relatively cheap for most medium-sized companies, and it provides a fair level of protection against outlier accidents.
I'm aware that many people using cloud also ascribe to the 100% uptime mentality, but for most companies that is simply not needed. I mean even for Netflix or Amazon Prime Video, I wonder if 2 hours of unexpected downtime per year would really be enough to make anyone cancel their service. I myself at least have spent much more time than that trying to get HDCP graphics cards drivers, HDMI cables, and the stars to align so that the Netflix app will work with 4K HDR playback on my TV.
So yes, (your 2nd paragraph) I would knowingly accept that there are critical bottlenecks that are unknown and that could be triggered by severe traffic spikes. And most of my customers would be happy to accept that risk in exchange for the cost savings of not proactively fixing the issue.
And if you look at the overall state of software, it looks like pretty much every company is happy to trade reliability/resilience for cost savings these days. That's why I applaud the efforts in the original article, but the pragmatic way seems to be to just skip the whole thing.
It does seem like a fad. Kudos to those who can create a new field and profit from it. On the one hand, "chaos engineering" seem a bit like "we don't understand our architecture well enough to know what its failure modes are, so let's just poke it and see what happens," but on the other hand, it seems at least a little bit analogous to fuzzing, which is certainly a technique that yields useful results that would have otherwise been overlooked until it was too late.
My first instinct was to agree with this, but from my experience it's extremely difficult to properly communicate failure modes 100% of the time across different teams in very large organizations. Dependencies that are fuzzy arise for example when a service A proxies data for client service B from some other service C. It doesn't help that the organization of teams in a company often severs lines of communication between teams who explicitly don't have dependencies but implicitly do. As a result, information gets lost in the process. Having a last line of defense in the form of a "chaos engineering" team may actually be the natural response of large organizations to counter the inherent messiness that is produced as a result of bureaucracy.
That and additionally it has implications for the development team as well. Using "chaos engineering" shifts the mindset of the developers. As a developer you now expect things to fail. You know that the "we make it work first and make it resilient later" approach will bite you sooner then later so you think resilience from the first line of code.
There are a million different ways a computer can fail. I think we're asking too much of people to be able to know all the pitfalls of every system they create.
But also this 'new field' just seems like something we've already been doing just with a different name. You're kind of expected to make sure your system can work if the computer suddenly shuts off, or a dependency is lost, or the network is slow. Have we not been doing this??
If nothing else, the article showcases how to use the recently released AWSSSMChaosRunner[0], which I hadn't heard about. Beyond that, it's good when multiple different high quality blog posts show how to do what's fundamentally the same thing.
Definitely seems like a case of someone that wanted to publish a blog post for some internal kudos, but I'm not complaining. Always good to share knowledge.
Never thought I’d see anti-article sentiment on HN, and more generally, not sure what sort of response you expect to generate. Why comment at all?
To avoid more of the same, I’ll say that the article itself is pretty detailed, but whereas Netflix blog-posts are more general, this one feels hyper-specific. That’s great if you’re tied to aws, but I think the former has longer-lasting + further-reaching utility.
Not an anti-article sentiment, but I dislike the trend of big companies pretending they care about the nerds just because they realize they need them for one particular objective.
That makes more sense. I think it’s worth framing differently, though.
Big companies aren’t writing the doc, per se. Engineers interested in the material are. And often, they want the opportunity to do so - it’s something of a stamp for “expertise” in their favor.
Assuming the post is actual quality, it’s a marketing win for the company, a professional/developmental win for the employee, and an informational win for everyone else.
We tried a similar chaos tool in our company built in-house. Simulated most of the scenarios mentioned here using SSM/other scripts. At first everyone was interested and after some time the interest faded. Our problem was lack of visualization across the app ecosystem i.e how will it impact the app ecosystem when a batch of ec2 instances are suddenly spiking on CPU and what will be the impact to end user.
Turns out people care only if there is an end user impact and doesn't really care about random anomalies.
And to build the capabilities required for measuring the impact + automating the workflow of the actual chaos tests is a lot of work
A library like AWSSSMChaosRunner would be a core component of building gameday like capability. But building a full gameday framework is out of the scope of this discussion.
You can do Chaos Engineering quite easily with Ruby, because you can raise an exception in a thread from another thread. Many years ago I built a simple tool which allowed you to specify a series of exceptions, and their frequency. The library would run up an extra thread in your process, and simply drop bombs (i.e. raise exceptions according to the required distribution) across the other threads at random.
It worked a treat for ensuring high availability in IoT systems
Always start out streaming in really, really low quality on the Firestick and gets stuck there despite us having gigabit fiber. Usually have to stop playing, exit and then go back in again then it works and plays in HD. Never happens on Netflix or Hulu or any other app, just Prime Video.
One thing I wonder (and find difficult to simulate) are failures in external services. Sure you can unit test your function with a 500 for example, but you never know in which ways the function/library can fail
(Not to mention the cases where it says everything worked but it didn't)
The article does talks about how to inject latency or packet-loss into calls to particular external services. This should help you test many service->dependency failure scenarios around retries, timeouts and circuit-breakers.
Injecting specific error codes or exceptions is a bit more complicated but it is possible with other approaches, for example: Chaos toolkit.
I have been getting failed to load error almost every week on Prime Video when I try to load a series at dinner time on weekdays (pacific timezone). Never have this issue with Netflix.
From my understanding you don’t technically own most media (music, movies, software) regardless of the format. It’s a license. Even if you purchase a CD or DVD, etc.
That misses the point. I have DVDs that I purchased that will work as long as I posses them and they were made before Netflix was even a company.
There may technically be a license attached to them, but there is no practical way for any company to revoke my usage of them and the failure mode if any of those companies cease to exist is for them to continue working.
No, that's nonsense. You do own the medium. More relevantly, the first sale doctrine means copyright does not restrict most of the ownership rights you'd expect to have.
I have — in this case Better Call Saul as I’ll probably watch it again in a few years. Since I don’t have to carry around a DVD I don’t have any cost of carrying around a DVD that I will probably lose.
I liked that Amazon reduced the price by almost the amount of the first two videos that I rented.
They have (in my opinion) an extremely good library of articles in their so called builders library.
For example these two below articles.
https://aws.amazon.com/builders-library/avoiding-fallback-in... https://aws.amazon.com/builders-library/leader-election-in-d...
These topics are extremely hard to solve from scratch yet they are distilled pretty well in the above articles and they include a further reading section.
I would implore others to have a gander. I wish the same could be said for their documentation.