about summary refs log tree commit diff
AgeCommit message (Collapse)AuthorFilesLines
2024-04-29 r/8026 refactor(tvix/glue): move decompression into fetchers/ subdirFlorian Klink3-5/+7
This is specifically used for the fetcher code (only). Moving it to there for now. Change-Id: I1e1d0541b85340ef4ff3a4c6b3fa99b51853f539 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11532 Reviewed-by: edef <edef@edef.eu> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-28 r/8025 chore(3p/sources): Bump channels & overlayssterni6-24/+27
- //tvix: address new clippy lints - //users/tazjin: Satisfy gonic module's new need for a playlist folder. - //users/aspen/games: adjust for changed location of df's default init.txt and d_init.txt. Change-Id: I00a2adb506ae866206fb6f88c39c9a6af320380f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11509 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI Reviewed-by: aspen <root@gws.fyi>
2024-04-28 r/8024 feat(tools/depot-deps): include josh-filtersterni1-0/+1
This is occasionally useful for testing //views or preparing new subtrees or subtree updates. Change-Id: I6841e5c3d105406fb960b59e16019f8444da238b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11458 Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2024-04-28 r/8023 refactor(3p): use josh from nixpkgssterni4-52/+3
The change we need has been released and propagated to nixos channels. Change-Id: Ib10a1d42d7ef6deaf5665a13b72ece345e83d7dc Reviewed-on: https://cl.tvl.fyi/c/depot/+/11457 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2024-04-28 r/8022 chore(tazj.in): add a convenience redirectVincent Ambo1-0/+5
Change-Id: Ia41fbff390a2b1df0926ab33e9f4f66b1fd92512 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11533 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2024-04-27 r/8021 feat(sterni/machines/ingeborg): switch to quasselsterni3-1/+19
Change-Id: I2970f47e22e6984db49d455b31903814fa94ec76 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11529 Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2024-04-27 r/8020 chore(3p/rust-crates:libgit2-sys): 0.16.1+1.7.1 -> 0.16.2+1.7.2sterni1-2/+2
Change-Id: I21ab23177cbbc903eae02ea95dc3fe51f731f777 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11528 Tested-by: BuildkiteCI Autosubmit: sterni <sternenseemann@systemli.org> Reviewed-by: sterni <sternenseemann@systemli.org>
2024-04-26 r/8019 feat(tvix/cli): change log format to compact, add RUST_LOG supportFlorian Klink1-1/+8
This brings feature parity with the tvix-store CLI command, at least as far as logging (not tracing) is concerned. Alternative to cl/11482, RUST_LOG seems to be the more canonical way to influence this, and is consistent with how it's done in tvix-store. Change-Id: I923a0b0ae55dc49af7efdacdcf5b1f24e561b3c2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11527 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-26 r/8018 feat(tvix/store/bin): disable --json option, set log to compactFlorian Klink1-30/+10
This wasn't really used - to ingest logs in meachine-readable form, the OTLP infrastructure is more suitable to provide structured logs than parsing JSON from std{err}, as it also captures span information. Also, the non-JSON output is a bit too spammy, as remarked in cl/11483 - change it to `compact`. Change-Id: I48007b84ba076ab566abbb6131a02868fe0eb397 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11526 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-26 r/8017 chore(nix-compat/nar/wire): space Tag::Buf docs properlyedef1-0/+1
Change-Id: If15b14b5328b9e7f85ea3fa7644d327dfb50e1a6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11523 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-26 r/8016 feat(nix-compat/wire/bytes/reader): split out reading the trailer blockedef1-56/+134
We separate ingesting the trailer block into a Future of its own, parametrised on the specifics of the trailer pattern. This is intended to be used for future work on an async NAR reader, which needs to read a terminating parenthesis as well as the regular padding. Thanks to @griff for suggesting separating the ingestion into its own Future. Change-Id: I36c2503baa67937046a63e9bf0cfc38201394025 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11522 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-26 r/8015 refactor(nix-compat/wire/bytes/reader): drop random whitespaceedef1-2/+2
Change-Id: Ic683eab435576acc8f7e03f5684767ffa468851a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11521 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-04-26 r/8014 refactor(nix-compat/wire/bytes/reader): drop Pin::get_mutedef2-4/+4
Pin<&mut T> is DerefMut when T: Unpin, so we don't actually need to explicitly call get_mut. Change-Id: Iaa312ec49c87100010e09c94f319e57e31da0cd5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11520 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-04-25 r/8013 refactor(nix-compat/wire): move BytesPacketPosition into writeredef2-20/+17
We don't use it in the reader anymore. Change-Id: I98fe204a747711464e9e7ca17df06fa9854eb344 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11519 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-25 r/8012 fix(nix-compat/bytes): make BytesReader less hazardousedef2-178/+317
We now *never* return the final bytes until we've read the padding in full, so read_exact is safe to use. This is implemented by TrailerReader, which splits the phases of reading (and validating) the final 8-byte block, and providing the contained payload bytes to the caller. Change-Id: I0d05946a8af9c260a18d71d2b763ba7a68b3c27f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11518 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-04-25 r/8011 feat(nix-compat/wire/bytes): allow specifying a pre-read sizeedef1-0/+41
Change-Id: I9c94239c308cfbc2e6dae871ba77fb33507433c9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11517 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-04-25 r/8010 refactor(nix-compat/wire/bytes): drop pin_project, clean upedef1-63/+57
We already require R: Unpin in the constructor, so there's not much use to pin projection. Change-Id: Ia7bf734dc3aa86ffa6d1d5de778939baa9676bb9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11516 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-04-25 r/8009 refactor(nix-compat/wire/bytes): don't parametrise on RangeBoundsedef1-12/+20
This is semantically a RangeInclusive, since we can only have 0..=u64::MAX at most, and monomorphising on the bounds doesn't buy us anything. Change-Id: Ib601d7fd77d703d6c8c5ec27ac9e67bb122ce1c0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11515 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-25 r/8008 fix(tvix): typoedef2-2/+2
Change-Id: Ibe4741b8086e9da442232c14cdb337556704cef6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11514 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-25 r/8007 test(tvix/nix-compat): add debug assertions for nar readertcmal2-84/+426
Adds debug assertions to ensure that the reader's variants are upheld. If any of the following happens, then the currently in use reader must be abandoned: * A directory or file reader encounters an error * A directory or file reader is dropped before being fully read from Additionally, a directory reader must not be read from again after it has returned None. These checks are only used when debug_assertions are on, so vanish in release mode. Resolves two TODO items added by edef Change-Id: I27bd9643a632798db5351957506c166b9bd5ca4e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11508 Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Autosubmit: Aria Shrimpton <me@aria.rip> Tested-by: BuildkiteCI
2024-04-25 r/8006 feat(ops/users): add caralice to userssterni1-0/+5
Change-Id: I3d907589c75939c86faa3c1276e4023126ad3d17 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11513 Autosubmit: sterni <sternenseemann@systemli.org> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2024-04-25 r/8005 docs(tvix): add a list of TODOs/ideasFlorian Klink2-0/+130
This contains a rough collection of ideas on the TODO list, trying to keep track of it somewhere. Change-Id: Ifc5b0cf9f7ac38f7a8e56515882bdf70e349544b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11512 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-24 r/8004 docs(tvix/nix-compat/wire/bytes/reader): fix typoFlorian Klink1-1/+1
This is a ReadBuf, not a BufRead. Change-Id: Ie80e894f4b24b77cdd60409ddfaa66dae0ffeec9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11511 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2024-04-24 r/8003 docs(tvix/nix-compat): add missing reference to BytesReaderFlorian Klink1-2/+2
Change-Id: Ideed83d191b55e131720e598b7591e8375a26cfd Reviewed-on: https://cl.tvl.fyi/c/depot/+/11510 Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu> Autosubmit: flokli <flokli@flokli.de>
2024-04-24 r/8002 refactor(tvix/castore): add separate Error enum for archivesConnor Brewster3-33/+37
The `Error` enum for the `imports` crate has both filesystem and archive specific errors and was starting to get messy. This adds a separate `Error` enum for archive-specific errors and then keeps a single `Archive` variant in the top-level import `Error` for all archive errors. Change-Id: I4cd0746c864e5ec50b1aa68c0630ef9cd05176c7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11498 Tested-by: BuildkiteCI Autosubmit: Connor Brewster <cbrewster@hey.com> Reviewed-by: flokli <flokli@flokli.de>
2024-04-23 r/8001 feat(tvix/castore): upload blobs concurrently when ingesting archivesConnor Brewster2-12/+92
Ingesting tarballs with a lot of small files is very slow because of the round trip time to the `BlobService`. To mitigate this, small blobs can be buffered into memory and uploaded concurrently in the background. Change-Id: I3376d11bb941ae35377a089b96849294c9c139e6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11497 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: Connor Brewster <cbrewster@hey.com>
2024-04-23 r/8000 refactor(tvix/castore): switch to `ingest_entries` for tarball ingestionConnor Brewster3-128/+239
With `ingest_entries` being more generalized, we can now use it for ingesting the directory entries generated from tarballs. Change-Id: Ie1f7a915c456045762e05fcc9af45771f121eb43 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11489 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-04-23 r/7999 fix(users/picnoir/tvix-daemon): gitignore target dir, use cleaned srcFlorian Klink2-2/+4
This prevents the nix build copying the target/ dir into the store whenever this is built through Nix. Change-Id: I397228fd8e2e3265ed87d3400fe927bc505da090 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11496 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-23 r/7998 fix(tvix/glue/tvix_store_io): remove early returnFlorian Klink1-157/+166
Doing the fetch comes up with the root node, but we still need to descend from there to the desired subpath. Move things around to ensure the fetch case also only sets root_node. This logic should probably be moved into smaller, easier to consume functions. Change-Id: I6ab9317df794f53d2504029bbc77859e89fef1ed Reviewed-on: https://cl.tvl.fyi/c/depot/+/11507 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-23 r/7997 fix(tvix/glue/fetchers): rename node name for all three typesFlorian Klink1-6/+2
We also need to rename the node in case it's a directory or symlink at the root. Change-Id: I6e9957200f65991645ae3e1755b943200453dfd5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11506 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-04-23 r/7996 feat(tvix/glue/fetchers): support file:// URLsFlorian Klink9-37/+124
Nix supports file:// - URLs for `fetchurl` and `fetchTarball`. Convert the enums and function arguments to hold a URL type. reqwest::Url is a re-export of the url crate, but they don't re-export the parsing errors, and as we want to hold these in our Error types, add it to Cargo.toml explicitly. The Fetcher::download function now checks on the scheme, and either opens the file locally, or does do a HTTP request as before. Fetch gets its custom debug impl, removing potentially sensitive username and password out of URLs. Change-Id: I777db1fe487370e822cbfec4624034aca5e08045 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11504 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-04-23 r/7995 test(tvix/glue): add tests for fetchurl and fetchTarballFlorian Klink4-0/+67
Change-Id: I53a0590ecf4e5fcb1bfd1d127824211338e28256 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11503 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-23 r/7994 feat(tvix/glue/store_io): have KnownPaths track fetches tooFlorian Klink5-19/+147
Have fetcher builtins call queue_fetch() whenever they don't need to fetch something immediately, and teach TvixStoreIO::store_path_to_node on how to look up (and call ingest_and persist on our Fetcher). Change-Id: Id4bd9d639fac9e4bee20c0b1c584148740b15c2f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11501 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-04-23 r/7993 refactor(tvix/glue): move Fetch[er] into its own types, fetch lazilyFlorian Klink8-392/+531
We actually want to delay fetching until we actually need the file. A simple evaluation asking for `.outPath` or `.drvPath` should work even in a pure offline environment. Before this CL, the fetching logic was quite distributed between tvix_store_io, and builtins/fetchers.rs. Rather than having various functions and conversions between structs, describe a Fetch as an enum type, with the fields describing the fetch. Define a store_path() function on top of `Fetch` which can be used to ask for the calculated store path (if the digest has been provided upfront). Have a `Fetcher` struct, and give it a `fetch_and_persist` function, taking a `Fetch` as well as a desired name, and have it deal with all the logic of persisting the PathInfos. It also returns a StorePathRef, similar to the `.store_path()` method on a `Fetch` struct. In a followup CL, we can extend KnownPaths to track fetches AND derivations, and then use `Fetcher` when we need to do IO into that store path. Change-Id: Ib39a96baeb661750a8706b461f8ba4abb342e777 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11500 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-22 r/7992 feat(tvix/boot): default CH_NUM_CPUS=2, and set num_queues= to itFlorian Klink1-3/+3
See https://patchwork.kernel.org/project/qemu-devel/patch/20200706135650.438362-5-stefanha@redhat.com/ Change-Id: I3ba9a1c1f8b0be27f215da6b71cd3a0ded513a75 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11502 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-22 r/7991 fix(tvix/eval): don't impl From<NixString> for StringFlorian Klink1-5/+1
This caught me by accident in an earlier revision of cl/11500 - I had a `NixString`, wanted to return it as a `String`, so I was naively calling `s.into()`. That unfortunately gave me the `Display` implementation of `NixString`, which quotes strings, causing an annoying error further up the stack. NixStrings are bytes, we can keep the impl From<NixString> for BString, but having a `.into()` suddenly do quoting is more than unexpected. Change-Id: I5434ba94bfe6c493d0a57e68225ecc22daa4b948 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11505 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-04-22 r/7990 docs(nix/buildkite): update error messageFlorian Klink1-1/+1
With custom phases, using `prompt` is not limited to the "release" phase only, we only care about it not being in the "build" phase. Change-Id: I8af27016865c728125dbe144d31006915b20b105 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11499 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2024-04-22 r/7989 refactor(tvix/cli/default.nix): make eval tests and benches derivationsFlorian Klink1-49/+45
This makes the eval tests and benchmarks standalone Nix derivations and readTree targets: ``` nix-build -A tvix.cli.eval-nixpkgs-cross-hello-outpath nix-build -A tvix.cli.benchmark-nixpkgs-attrnames ``` Even without doing any fetches, We need to set `SSL_CERT_FILE`, so reqwest is able to load its CA roots. Change-Id: Ib45282d01044165c7816391adbeeb26334f8e924 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11493 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-04-20 r/7988 feat(tvix/store/bin): upload paths concurrently in `copy`Florian Klink1-15/+53
We can speedup uploads further by not uploading all store paths sequentially, but in parallel. We still don't respect the reference graph, so nothing changed here. Change-Id: I40edb5725fe7a15170f5b9f8f4c8bec2130ecca3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11492 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-04-20 r/7987 refactor(tvix/castore): ingest filesystem entries in parallelFlorian Klink4-82/+53
Rather than carrying around an Future in the IngestionEntry::Regular, simply carry the plain B3Digest. Code reading through a non-seekable data stream has no choice but to read and upload blobs immediately, and code seeking through something seekable (like a filesystem) probably knows better what concurrency to pick when ingesting, rather than the consuming side. (Our only) one of these seekable source implementations is now doing exactly that. We produce a stream of futures, and then use [StreamExt::buffered] to process more than one, concurrently. We still keep the same order, to avoid shuffling things and violating the stream order. This also cleans up walk_path_for_ingestion in castore/import, as well as ingest_dir_entries in glue/tvix_store_io. Change-Id: I5eb70f3e1e372c74bcbfcf6b6e2653eba36e151d Reviewed-on: https://cl.tvl.fyi/c/depot/+/11491 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-04-20 r/7986 fix(tvix): fix outdated comment and error in TvixStoreIO::openConnor Brewster1-3/+3
This function was originally called `read_to_string` but was changed to `open` to make it so that file contents aren't always held in memory. A comment and error message were not updated to reflect the new name of this method. Change-Id: I3d86e2f6d7006c2e1513121fc3c62efcb7e7b9bb Reviewed-on: https://cl.tvl.fyi/c/depot/+/11495 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-20 r/7985 feat(tvix/castore): Fix build warnings in release modeConnor Brewster2-0/+2
Fixes some build warnings that only happen when building in release mode which disables `debug_assertions`. Change-Id: I554d5fce7c869c23cf4aa93179f0ee9f7f7c834e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11490 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: Connor Brewster <cbrewster@hey.com> Reviewed-by: flokli <flokli@flokli.de>
2024-04-20 r/7984 fix(tvix/castore): ensure all directories are present during ingestionConnor Brewster1-0/+8
`ingest_entries` requires that all directories referenced by entries in the ingestion stream have an explicit entry in the stream. For example, if the stream contains a file with path `foo/bar`, there must be an entry that comes later in the stream for the directory `foo`. Change-Id: I61b4fbbb73ea7278715e04271d8073b484e05e61 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11488 Autosubmit: Connor Brewster <cbrewster@hey.com> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-20 r/7983 feat(tvix/eval): Implement builtins.fetchTarballAspen Smith10-33/+306
Implement a first pass at the fetchTarball builtin. This uses much of the same machinery as fetchUrl, but has the extra complexity that tarballs have to be extracted and imported as store paths (into the directory- and blob-services) before hashing. That's reasonably involved due to the structure of those two services. This is (unfortunately) not easy to test in an automated way, but I've tested it manually for now and it seems to work: tvix-repl> (import ../. {}).third_party.nixpkgs.hello.outPath => "/nix/store/dbghhbq1x39yxgkv3vkgfwbxrmw9nfzi-hello-2.12.1" :: string Co-authored-by: Connor Brewster <cbrewster@hey.com> Change-Id: I57afc6b91bad617a608a35bb357861e782a864c8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11020 Autosubmit: aspen <root@gws.fyi> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-20 r/7982 feat(tvix/castore/import): only allow normal components in entry pathsFlorian Klink1-1/+10
Explicitly document and add a debug assertion for that. It's up to callers to ensure this doesn't happen. Change-Id: Ib5d154809c2ad2920258e239993d0b790d846dc8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11487 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-20 r/7981 refactor(tvix/castore/import): make module, split off fs and errorFlorian Klink7-197/+225
Move error types and filesystem-specific functions to a separate file, and keep the fs:: namespace in public exports. Change-Id: I5e9e83ad78d9aea38553fafc293d3e4f8c31a8c1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11486 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2024-04-20 r/7980 refactor(tvix/castore/import): rename ingest_entries argFlorian Klink1-6/+3
This is not a stream of direntries anymore, but a stream of ingestion entries. Change-Id: I387f4497b6567066b24c58ca0262e710348180e9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11485 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-19 r/7979 refactor(tvix/castore): generalize store ingestion streamsConnor Brewster5-258/+256
Previously the store ingestion code was coupled to `walkdir::DirEntry`s produced by the `walkdir` crate which made it impossible to reuse ingesting from other sources like tarballs or NARs. This introduces a `IngestionEntry` which carries enough information for store ingestion and a future for computing the Blake3 digest of files. This allows the producer to perform file uploads in a way that makes sense for the source, ie. the filesystem upload could concurrently upload multiple files at the same time, while the NAR ingestor will need to ingest the entire blob before yielding the next blob in the stream. In the future we can buffer small blobs and upload them concurrently, but the full blob still needs to be read from the NAR before advancing. Change-Id: I6d144063e2ba5b05e765bac1f27d41b3c8e7b283 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11462 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-19 r/7978 feat(tvix/castore): add convenience `add` method to DirectoryConnor Brewster2-1/+166
This adds `Directory::add` which is a convenience helper for adding nodes into a `Directory` while preserving sorted order. This implements `Ord` and `PartialOrd` for `FileNode`, `SymlinkNode`, and `DirectoryNode` so `binary_search` can be used. Change-Id: I94b86bdef5d0da55aa352e098988b9704cafca19 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11481 Autosubmit: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-04-19 r/7977 chore(tvix/glue): migrate from test_case to rstestFlorian Klink5-234/+218
Change-Id: Id1bac94e7b95960dabd55f095f04bdabfeee10ea Reviewed-on: https://cl.tvl.fyi/c/depot/+/11476 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI