A119: Slicer LB Policy - #551
Conversation
|
@markdroth @ejona86 @dfawley |
|
I'm going to be making some structural changes to the spec based on our review today. Will ping here once it is ready for review. @markdroth @ejona86 @dfawley @shivaspeaks @pawbhard Please expect some changes to the spec, but not wholesale changes. So, you could start looking at it from an implementation pov. Thanks. |
|
The PR is ready to be looked at. |
markdroth
left a comment
markdroth
left a comment
There was a problem hiding this comment.
This looks really good!
I have a lot of comments here, but they're mostly fine details or clarifications -- the high-level shape of this is solid, and the doc does a good job of describing it.
Please let me know if you have any questions. Thanks!
markdroth
left a comment
markdroth
left a comment
There was a problem hiding this comment.
This is getting closer!
Please let me know if you have any questions. Thanks!
| // A timeout value for fallback to kick in when no assignments have been | ||
| // received from the sharding service. | ||
| // Defaults to 60 seconds if not specified. | ||
| google.protobuf.Duration fallback_timeout = 6; |
There was a problem hiding this comment.
It doesn't really matter whether it's defined here or in A121. I guess it should be in whichever of the two gRFCs is going to be merged last.
Please coordinate with @mbissa.
| allEndpointsInSlice []int // Indices into sliceMap.allEndpoints | ||
| endpointsByState [5][]int // Array indexed directly by connectivity.State (ranges 0..4) |
There was a problem hiding this comment.
Okay. @pawbhard, let's consider this for the C-core implementation.
| 1. Parsing a `GrpcService` proto embedded within an LB policy's configuration | ||
| into its internal representation, requires access to the following: | ||
| * the complete bootstrap configuration to access the `allowed_grpc_services` | ||
| section of the bootstrap configuration. | ||
| * configuration of the specific xDS server that delivered this resource, to | ||
| determine if the server is to be trusted or not. |
| * Sharding services usually move endpoints frequently between key-ranges. LB | ||
| policies that maintain scheduling state apart from endpoint state (like WRR) | ||
| would have to reset their scheduling state, thereby making them less | ||
| effective. |
There was a problem hiding this comment.
Wouldn't the slicer refresh rate usually be significantly larger than the WRR refresh rate (default to 1m iirc). Because the slicer likely requires starting new instances, moving data around, etc.
There was a problem hiding this comment.
I'm not sure what refresh rate you're referring to for WRR here. If you mean the weight_update_period, that actually defaults to 1s, not 1m. But I'm not sure that's actually relevant here.
I don't think we can assume any particular refresh rate for the slicer service. It could very well be updated every few seconds -- that's determined by the slicer service, not by the client, so we can't make any assumptions here.
Regardless of refresh rate, though, the WRR policy won't even use the weight data for an endpointt until after the blackout period, which is 10 seconds by default -- which means that any endpoint that gets added to a slice will wind up using average weight data for the first 10 seconds it's there. I think we are expecting to see enough movement of endpoints between slices that this will significantly break the load distribution that WRR normally provides.
There was a problem hiding this comment.
I'm not sure about any general case but specifically for our use case we'd be moving endpoints within a slice on the order of 10s of minutes to hours, while orca weight changes on every request.
Really my overall point in my comments is that the current proposal makes it impossible to combine the slicer and wrr, while there are valid use cases where combine the 2 is beneficial. I am fine with the default child LB being whatever, so long that there is an option to configure one that makes more sense for specific use cases
|
|
||
| ## Rationale | ||
|
|
||
| ### Why not create a child LB policy for every `SliceEntry`? |
There was a problem hiding this comment.
Here is an example use case we have at Reddit for this. A service caches a significant dataset in local memory, but this will soon be too big to fit on a single instance, so we were hoping to define shards/slices. Each instance gets assigned to a few slices, preload the data and start serving.
But also, not all instances have the same capacity due to a mix of multi-tenancy and mixed hardware. So if we randomly load balance we end up with instances that are severely overloaded while others have spare capacity. We've been using orca + wrr to address this issue with great success.
An alternative for us would be to define an xds cluster per slice and route based on a header but this feels like a poor man's version of this proposal.
There was a problem hiding this comment.
Having endpoints with different capacities makes the problem much harder, and I'm not sure that's something we can easily solve in the general case. Even if you did use WRR for something like this, I think it could potentially lead to weird feedback loops. The slicer server is going to be moving keys around based on load, but WRR could be skewing the load seen by the endpoints, so you might wind up with weird oscillations.
The best thing I can suggest is to have the slicer server take the endpoints' capacities into account when assigning slices to endpoints to provide better balancing. The slicer server is already going to be responsible for looking at the endpoints' load, so it doesn't seem unreasonable to expect it to understand this.
There was a problem hiding this comment.
WRR and the slicer solve for 2 problems with very different timescales though. Yes the slicer will need to be aware of load averages to decide of the shard placement, but it can't react to things like CPU pressure which WRR does very well.
I'm not saying this should be default behavior, but having the option to nest WRR until this new slicer LB does solve a real problem for us.
|
@markdroth : I think I've addressed all your comments. Thanks for the detailed review. |
| class PickerEndpoint: | ||
| state: ConnectivityState | ||
| picker: Picker | ||
| child_lb: Balancer # Child balancer to request a connection to the endpoint |
There was a problem hiding this comment.
In C-core, LB policies are single-owner, so we can't hold a ref to it here. We will instead need to hold a ref to the EndpointState object, which is the single owner of the child policy.
There was a problem hiding this comment.
Do you prefer that I change it to how you had in your pseudo-code? I thought this was simpler and this is totally possible in Go (but in practice, we will store an interface that only has the ExitIdle method and not the complete Balancer interface). And I'm guessing this is possbile in Java as well.
There was a problem hiding this comment.
Yeah, sounds like this is an area where there will be differences between languages. I'm not worried about the details here, but I do want the intent of the design to be clear. My concern with having this be of type Balancer is that @pawbhard (who happens to be the one implementing this in C-core) may look at this and get confused because that won't work in that implementation. :)
Maybe encode it as something like Balancer | EndpointState? We can add a comment that explains that the purpose of the field is to alloww the picker to trigger a connection attempt on the child policy, and that the details will differ by language. We can even explicitlly say that implementations might use some other interface if that's more appropriate.
| If validation fails, the LB policy must terminate the stream to the sharding | ||
| service, and attempt to re-establish it. |
There was a problem hiding this comment.
Is this really the right thing to do? If we get an invalid assignment, that's likely a bug in the slicer service, and it's not clear to me that reestablishing the stream will fix it -- we'll probably just get the same bad data again after we restart the stream. But restarting the stream will cause load on the server, so I'd be worried about a cascading failure here: a bug in the slicer service could cause all of the clients to start terminating and reconnecting streams, which could cause a lot of unexpected load.
I suggest asking the slicer folks how they handle this kind of case today.
|
|
||
| def pick(self, pick_args: PickArgs) -> PickResult: | ||
| # Extract sharding key from request metadata/header | ||
| key = extract_key_from_metadata(pick_args, self.slice_key_header_name) | ||
| key = extract_key_from_metadata(pick_args, self.lb_config.slice_key_header_name) | ||
|
|
||
| # Lookup matching slice range index and SliceEntry | ||
| slice_idx, slice_entry = self.slice_map.lookup(key) |
There was a problem hiding this comment.
This now returns only the index, not the slice entry.
| class PickerEndpoint: | ||
| state: ConnectivityState | ||
| picker: Picker | ||
| child_lb: Balancer # Child balancer to request a connection to the endpoint |
There was a problem hiding this comment.
Yeah, sounds like this is an area where there will be differences between languages. I'm not worried about the details here, but I do want the intent of the design to be clear. My concern with having this be of type Balancer is that @pawbhard (who happens to be the one implementing this in C-core) may look at this and get confused because that won't work in that implementation. :)
Maybe encode it as something like Balancer | EndpointState? We can add a comment that explains that the purpose of the field is to alloww the picker to trigger a connection attempt on the child policy, and that the details will differ by language. We can even explicitlly say that implementations might use some other interface if that's more appropriate.
3 participants
No description provided.