Skip to content

Regenerate the protobuf bindings - #140

Open
Lumen951 wants to merge 1 commit into
google:masterfrom
Lumen951:regen-protobuf-bindings
Open

Lumen951 wants to merge 1 commit into
google:masterfrom
Lumen951:regen-protobuf-bindings

Conversation

@Lumen951

Copy link
Copy Markdown

While getting the package importable so I could follow the inference path, I found the
checked-in protobuf bindings do not load. On master all five generated modules are
unimportable, for two different reasons:

ffn.utils.vector_pb2                 TypeError: Descriptors cannot be created directly.
ffn.utils.bounding_box_pb2           TypeError: Descriptors cannot be created directly.
ffn.inference.inference_pb2          TypeError: Descriptors cannot be created directly.
ffn.inference.consensus_pb2          ModuleNotFoundError: No module named 'utils'
ffn.inference.resegmentation_pb2     ModuleNotFoundError: No module named 'utils'

The first three were generated by a protoc from the Python 2 era: they still carry the
_b = sys.version_info[0] < 3 and ... shim and contain no
_runtime_version.ValidateProtobufRuntimeVersion call, so protobuf >= 3.19 refuses to
load them.

The other two fail before they get that far, and would fail on any protobuf version,
because their generated imports are not package-relative:

from utils import vector_pb2 as utils_dot_vector__pb2
from inference import inference_pb2 as inference_dot_inference__pb2

Those only resolve when ffn/ itself is on sys.path, not when the package is
imported as ffn.*. Note that the two files that do it right use the other form, so
the checked-in generated files are not even consistent with each other.

Regenerating from the checked-in sources does not fix this on its own, because the
sources and the generated files disagree about the include root. The .proto files say

import "utils/vector.proto";

so generating with -I ffn follows them literally and reproduces the unimportable
from utils import ... form. Prefixing those imports with ffn/ and generating from
the repository root makes the sources and the generated output agree, and the output
then carries the runtime version guard, so a future protobuf release fails with a clear
message instead of the "Descriptors cannot be created directly" one.

Regenerated with:

python -m grpc_tools.protoc -I. --python_out=. \
  ffn/utils/vector.proto ffn/utils/bounding_box.proto \
  ffn/inference/inference.proto ffn/inference/consensus.proto \
  ffn/inference/resegmentation.proto

ffn/utils/vector.proto has no imports of its own, so it is unchanged. The Apache
header of each generated file was restored afterwards, since protoc replaces it with
its own banner.

Verification

Clean trees exported from master and from this branch, same interpreter
(protobuf 7.36.2):

before: 3 x TypeError, 2 x ModuleNotFoundError
after:  all five import

One thing to decide

These were generated with protoc 7.35.1, which is what I had to hand. The guard
requires the gencode and the runtime to share a major version, so taking this would
require protobuf 7.x at runtime. If you would rather not move the floor that far, say
the word and I will regenerate with an older protoc (4.x or 5.x); nothing else in the
diff would change.

Found while trying to bring the package up from a clean checkout: every module
that touches a proto failed with "Descriptors cannot be created directly". The
checked-in *_pb2.py files were produced by a protoc from the Python 2 era
(they still carry the `_b = sys.version_info[0] < 3 and ...` shim) and have no
runtime version check, so protobuf >= 3.19 refuses to load them.

Regenerating from the checked-in .proto files did not work either. Those
import "utils/vector.proto", which makes protoc emit

    from utils import vector_pb2 as utils_dot_vector__pb2

whereas the checked-in files use the package-absolute form

    from ffn.utils import vector_pb2 as utils_dot_vector__pb2

so the sources and the generated files disagree about the include root, and no
single include root reproduces what is checked in. Prefixing the imports with
ffn/ and generating from the repository root makes the two agree, and emits
the runtime version guard so a future protobuf release fails loudly instead of
confusingly.

Regenerated with:

  python -m grpc_tools.protoc -I. --python_out=. \
    ffn/utils/vector.proto ffn/utils/bounding_box.proto \
    ffn/inference/inference.proto ffn/inference/consensus.proto \
    ffn/inference/resegmentation.proto

ffn/utils/vector.proto is unchanged: it has no imports to rewrite. The Apache
header of each generated file was restored afterwards, as protoc replaces it
with its own banner.
@google-cla

google-cla Bot commented Sep 25, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant