c++ ninja build failing on gh-actions using windows-latest about tensorstore HOT 12 CLOSED

No, haven't seen that. I just submitted a fix for cmake with the make generator.

It looks like we have another build error at HEAD; I'll see if I can get a fix out for that one then you can check to see if HEAD works.

from tensorstore.

Ok, please try again. HEAD incorporates fixes for missing directories when using the make generator (22a2396); I think that the above command also failed in that case, and might be fixed.

from tensorstore.

I think I got an identical error: (full run is here: https://github.com/allen-cell-animated/agave/actions/runs/9455178148/job/26044360214)
I see a whole lot of "blind reference" and "unknown library" in the cmake bazel steps early in the logs. Could that be related? I've been seeing them for a while, and on working builds too.

[498/5250] Linking CXX static library _deps\protobuf-build\protobuf_upb_generator_file_layout_stage0.lib [499/5250] Generating src/google/protobuf/wkt/google/protobuf/any.pb.h, src/google/protobuf/wkt/google/protobuf/api.pb.h, src/google/protobuf/wkt/google/protobuf/duration.pb.h, src/google/protobuf/wkt/google/protobuf/empty.pb.h, src/google/protobuf/wkt/google/protobuf/field_mask.pb.h, src/google/protobuf/wkt/google/protobuf/source_context.pb.h, src/google/protobuf/wkt/google/protobuf/struct.pb.h, src/google/protobuf/wkt/google/protobuf/timestamp.pb.h, src/google/protobuf/wkt/google/protobuf/type.pb.h, src/google/protobuf/wkt/google/protobuf/wrappers.pb.h, src/google/protobuf/wkt/google/protobuf/any.pb.cc, src/google/protobuf/wkt/google/protobuf/api.pb.cc, src/google/protobuf/wkt/google/protobuf/duration.pb.cc, src/google/protobuf/wkt/google/protobuf/empty.pb.cc, src/google/protobuf/wkt/google/protobuf/field_mask.pb.cc, src/google/protobuf/wkt/google/protobuf/source_context.pb.cc, src/google/protobuf/wkt/google/protobuf/struct.pb.cc, src/google/protobuf/wkt/google/protobuf/timestamp.pb.cc, src/google/protobuf/wkt/google/protobuf/type.pb.cc, src/google/protobuf/wkt/google/protobuf/wrappers.pb.cc FAILED: _deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/any.pb.h _deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/api.pb.h _deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/duration.pb.h _deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/empty.pb.h _deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/field_mask.pb.h _deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/source_context.pb.h _deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/struct.pb.h _deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/timestamp.pb.h _deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/type.pb.h _deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/wrappers.pb.h _deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/any.pb.cc _deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/api.pb.cc _deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/duration.pb.cc _deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/empty.pb.cc _deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/field_mask.pb.cc _deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/source_context.pb.cc _deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/struct.pb.cc _deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/timestamp.pb.cc _deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/type.pb.cc _deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/wrappers.pb.cc D:/a/agave/agave/build/_deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/any.pb.h D:/a/agave/agave/build/_deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/api.pb.h D:/a/agave/agave/build/_deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/duration.pb.h D:/a/agave/agave/build/_deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/empty.pb.h D:/a/agave/agave/build/_deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/field_mask.pb.h D:/a/agave/agave/build/_deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/source_context.pb.h D:/a/agave/agave/build/_deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/struct.pb.h D:/a/agave/agave/build/_deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/timestamp.pb.h D:/a/agave/agave/build/_deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/type.pb.h D:/a/agave/agave/build/_deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/wrappers.pb.h D:/a/agave/agave/build/_deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/any.pb.cc D:/a/agave/agave/build/_deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/api.pb.cc D:/a/agave/agave/build/_deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/duration.pb.cc D:/a/agave/agave/build/_deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/empty.pb.cc D:/a/agave/agave/build/_deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/field_mask.pb.cc D:/a/agave/agave/build/_deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/source_context.pb.cc D:/a/agave/agave/build/_deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/struct.pb.cc D:/a/agave/agave/build/_deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/timestamp.pb.cc D:/a/agave/agave/build/_deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/type.pb.cc D:/a/agave/agave/build/_deps/protobuf-build/src/google/protobuf/wkt/google/protobuf/wrappers.pb.cc C:\Windows\system32\cmd.exe /C "cd /D D:\a\agave\agave\build\_deps\protobuf-src && D:\a\agave\agave\build\bin\protobuf_protoc.exe --cpp_out=dllexport_decl=PROTOBUF_EXPORT:D:/a/agave/agave/build/_deps/protobuf-build/src/google/protobuf/wkt --proto_path=src src/google/protobuf/any.proto src/google/protobuf/api.proto src/google/protobuf/duration.proto src/google/protobuf/empty.proto src/google/protobuf/field_mask.proto src/google/protobuf/source_context.proto src/google/protobuf/struct.proto src/google/protobuf/timestamp.proto src/google/protobuf/type.proto src/google/protobuf/wrappers.proto" [500/5250] Linking CXX static library _deps\absl-build\absl\random\absl_random_internal_randen_slow.lib

from tensorstore.

Lines such as this are expected in the cmake output; they basically signal that some bazel libraries are not yet known, and indicate that a "default" behavior may be used.

Unknown library: @com_google_disco_to_proto3_converter//rules_gapic:disco_to_proto.bzl
Blind reference to @com_google_protobuf//:descriptor_proto__cpp_library from PackageId.parse("@com_google_googleapis//google/api")

I suspect that the D:/ in the command is causing trouble:

--cpp_out=dllexport_decl=PROTOBUF_EXPORT:D:/a/agave/agave/build/_deps/protobuf-build/src/google/protobuf/wkt

Is there a specific change which triggered this (either in agave or tensorstore)? I'm not sure where the dllexport_decl comes from.

When tensorstore builds as a cmake project it generates output files. I might need to see this one: _deps/protobuf-build/build_rules.cmake

Edit:

So I think that this comes from our cmake translation of a bazel rule using $(RULEDIR) or similar.
In bazel_to_cmake/variable_substitution.py this is translated to an absolute path, but we could probably translate it to a subdirectory of ${PROJECT_BINARY_DIR} or ${CMAKE_CURRENT_BINARY_DIR}.

from tensorstore.

If you have a way to test this, you could try applying this to bazel_to_cmake/variable_substitutions.py:

diff --git a/tools/cmake/bazel_to_cmake/variable_substitution.py b/tools/cmake/bazel_to_cmake/variable_substitution.py
index 1cb5438..566ad50 100644
--- a/tools/cmake/bazel_to_cmake/variable_substitution.py
+++ b/tools/cmake/bazel_to_cmake/variable_substitution.py
@@ -306,28 +306,30 @@
   source_directory = _context.resolve_source_root(
       _context.caller_package_id.repository_id
   )
+  binary_directory = pathlib.PurePosixPath(
+      _context.resolve_output_root(_target.repository_id)
+  )
 
-  def _relative(path: str) -> pathlib.PurePath:
+  def _relative_to_src(path: str) -> pathlib.PurePath:
     nonlocal source_directory
     return pathlib.PurePath(os.path.relpath(path, str(source_directory)))
 
-  binary_dir = pathlib.PurePosixPath(
-      _context.resolve_output_root(_target.repository_id)
-  )
-  package_binary_dir = binary_dir.joinpath(_target.package_name)
+  def _relative_to_bin(path: str) -> pathlib.PurePath:
+    nonlocal binary_directory
+    return pathlib.PurePath(os.path.relpath(path, str(binary_directory)))
 
   relative_src_files = [
-      json.dumps(_relative(path).as_posix()) for path in src_files
+      json.dumps(_relative_to_src(path).as_posix()) for path in src_files
   ]
   quoted_out_files = [json.dumps(path) for path in out_files]
 
   substitutions: Dict[str, str] = {
-      "GENDIR": str(binary_dir),
-      "BINDIR": str(binary_dir),
+      "GENDIR": "${CMAKE_CURRENT_BINARY_DIR}",
+      "BINDIR": "${CMAKE_CURRENT_BINARY_DIR}",
       "SRCS": " ".join(relative_src_files),
       "OUTS": " ".join(quoted_out_files),
-      "RULEDIR": str(package_binary_dir),
-      "@D": str(package_binary_dir),
+      "RULEDIR": f"${{CMAKE_CURRENT_BINARY_DIR}}/{_target.package_name}",
+      "@D": f"${{CMAKE_CURRENT_BINARY_DIR}}/{_target.package_name}",
   }
 
   if len(src_files) == 1:
@@ -335,7 +337,10 @@
 
   if len(out_files) == 1:
     substitutions["@"] = quoted_out_files[0]
-    substitutions["@D"] = json.dumps(os.path.dirname(out_files[0]))
+    substitutions["@D"] = json.dumps(
+        "${CMAKE_CURRENT_BINARY_DIR}/"
+        + _relative_to_bin(os.path.dirname(out_files[0])).as_posix()
+    )
 
   return substitutions

from tensorstore.

BTW, the protobuf BUILD rule which corresponds to this command is:

https://github.com/protocolbuffers/protobuf/blob/main/src/google/protobuf/BUILD.bazel#L169

from tensorstore.

Thanks for giving this some attention.
I haven't yet been able to make time to try to add your patch within my cmake (frankly I'm not sure how to yet). It's odd that this is only happening on my github-actions with windows-latest. Locally my windows builds have been working.

Is there a way to run in a verbose mode that will output that D:/a/ directory path? I only see it when it errors.

from tensorstore.

Is there any way to get the gh-actions log file?

I wonder what happens if you build tensorstore on a local windows machine, then try to run the command by itself.
Something like:

D:\a\agave\agave\build\bin\protobuf_protoc.exe --cpp_out=dllexport_decl=PROTOBUF_EXPORT:D:/a/agave/agave/build/_deps/protobuf-build/src/google/protobuf/wkt --proto_path=src src/google/protobuf/any.proto src/google/protobuf/api.proto src/google/protobuf/duration.proto src/google/protobuf/empty.proto src/google/protobuf/field_mask.proto src/google/protobuf/source_context.proto src/google/protobuf/struct.proto src/google/protobuf/timestamp.proto src/google/protobuf/type.proto src/google/protobuf/wrappers.proto"

from tensorstore.

I saw the new 0.1.61 tag so gave it a try and my build went through on windows, mac and linux! So I'm content for now to close this!

FetchContent_Declare(
tensorstore
GIT_REPOSITORY https://github.com/google/tensorstore.git
GIT_TAG v0.1.61
)

from tensorstore.

Incidentally, is it considered worse to use GIT_REPOSITORY/GIT_TAG as opposed to URL/URL_HASH?

from tensorstore.

Ok, glad this worked. Apparently my concern about C:\ was wrong; instead it could have been that the output directory didn't exist.

As far as the CMake FetchContent issue, I don't know.

from tensorstore.

Since git tags are not guaranteed to be immutable you can't guarantee a reproducible build by specifying a tag. Specifying a commit hash would be sufficient for that, though.

from tensorstore.