about summary refs log tree commit diff
AgeCommit message (Collapse)AuthorFilesLines
2024-04-19 r/7960 feat(aspen/lusca): tweak alacritty font size; install slackAspen Smith1-1/+3
Change-Id: I14dd872fb3b6f495e57f3e538486c5fad901ccc2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11477 Autosubmit: aspen <root@gws.fyi> Tested-by: BuildkiteCI Reviewed-by: aspen <root@gws.fyi>
2024-04-19 r/7959 chore(nix/buildGo): remove unused functionFlorian Klink1-2/+0
Spotted by deadnix. Change-Id: Ia5a9ec86fe7bbfe1b4fa90dc135c000b429df334 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11461 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2024-04-19 r/7958 chore(nix/buildGo): cleanupsFlorian Klink1-4/+3
io/ioutil is deprecated, and the range expression can be simplified. Change-Id: I73b85991faafa333bddf90adbdefe2006f8d409c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11460 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-18 r/7957 chore(users/tazjin/keys): update yubikey key on khamovnikVincent Ambo1-1/+1
Change-Id: I9b8ca783a994e862788fea7b180541f5febeb7b5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11459 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2024-04-18 r/7956 fix(views/kit): include dependency-analyzersterni2-1/+11
Since cl/11116, //nix/buildkite needs dependency-analyzer as an input. Change-Id: I2d924d6648b4ebf39388d8142a99496ef854d0c1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11451 Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com> Tested-by: BuildkiteCI
2024-04-18 r/7955 feat(tvix/nix-compat/nix_daemon/version): add ProtocolVersionFlorian Klink5-27/+160
This provides a nice wrapper struct to deal with versions. Change-Id: I6acc03bc9f8d84a0583196073b52776c45d3fe92 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11454 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-18 r/7954 feat(tvix/nix-compat): enable more features by defaultFlorian Klink1-0/+3
There's no point in not enabling these, and rust-analyzer is quite useless too. nix-compat users outside of here can still explicitly disable default features if they don't want to pull in these features. Change-Id: I8f2f24e7734ad152d1733bc3421098fc91af290a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11455 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-18 r/7953 fix(ops/nixery-01): enable automatic GCVincent Ambo1-0/+9
This relates to nixery#167. Using our GC module is much more reliable than what we were doing previously. Change-Id: I1956457812a3a847a7c8a1f4e7e91e50fad08ac0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11453 Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de>
2024-04-18 r/7952 fix(ops): update DNS record for nixery-01Vincent Ambo2-2/+2
I accidentally deallocated the previous public IP, and had to make a new one :( Change-Id: Ie30305bdfdb8443e058270e5324baf555343441c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11452 Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de>
2024-04-16 r/7951 feat(tvix/boot): boot nixos VMFlorian Klink1-0/+34
This makes use of the vmCmdline and assertVMOutput to boot a NixOS VM off of `tvix-store virtiofs` have it write a message on successful bootup, then have it shut down again. We check we see the message in the output. Seeding the store with the closure it's what's currently taking most of the time (as it's all linear still), so log a message before booting the VM. Change-Id: I6887d935f589eb0ad6507033c01337e1ec500a0b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11191 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-04-16 r/7950 fix(tvix/castore/blobservice/grpc): don't use NaiveSeeker for nowFlorian Klink1-9/+17
Userland likes to seek backwards, and until we have store composition and can serve chunks from a local cache, we need to buffer the individual chunks in memory. Change-Id: I66978a0722d5f55ed4a9a49d116cecb64a01995d Reviewed-on: https://cl.tvl.fyi/c/depot/+/11448 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-16 r/7949 fix(tvix/castore/src): record rq.handle field in read()Florian Klink1-1/+1
This makes it easier to separate concurrent requests on the same inode. Change-Id: I7637c1d889336beeb0d186182ce22fbf60fd16c3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11447 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-04-16 r/7948 fix(tvix/castore/fs): use io::copy to fill kernel-provided bufferFlorian Klink1-3/+12
The docs state we must fill all of the buffer, except on EOF. Change-Id: Id977ba99c0b15132422474ebbf82bb92b79d55ba Reviewed-on: https://cl.tvl.fyi/c/depot/+/11446 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-16 r/7947 feat(tvix/castore/blob/naive_seeker): add some more tracingFlorian Klink1-10/+17
Change-Id: Iecf4a82a7d84008a8620825570b34e9094e6d590 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11445 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-04-16 r/7946 feat(tvix/castore/blob/chunked_reader): add some more tracesFlorian Klink1-2/+8
Change-Id: I2408707a7bc0e1c0cd8bd2933f8d68805b9e12c9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11444 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-16 r/7945 fix(tvix/castore/blobservice/chunk_rd): only skip *first* chunk bytesFlorian Klink1-3/+4
When (re)initializing a chunked reader, we were erroneously skipping the first n bytes from all chunks, not just the first one. Fix this, by passing in an enumerated list of chunks, and only calling SeekFrom::Start() on the first chunk in the stream. With this, I'm able to invoke b3sum on bin/bash successfully. Change-Id: I52ea480569267e093b0ac9d6bcd5c2d1b4db25f7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11443 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-04-16 r/7944 refactor(tvix/castore/blob/naive_seeker): rework skipping for clarityFlorian Klink1-42/+35
Increase the discard_buf to 4096 (as I've seen this size). Use the ready! macro to propagate pendings. Make it more clear what exactly should be skipped in total, and what during the current iteration. Also write down that poll_read call already takes care of updating self.pos, as I ran into that trap earlier (and added it here). Change-Id: I2d22e1c8a835c0f3dd0c648917009b2bad4fd57c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11442 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-04-16 r/7943 feat(tvix/castore/blob/chunked_reader): only reassemble on real seekFlorian Klink1-16/+16
If the resulting offset equals to our current position, there's no need to recreate a reader. Change-Id: I855f0c79c514c16ca48a78e12978af2835fbbd6a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11441 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-16 r/7942 docs(nix-1p): add (temporary) "ad" for Volga SprintVincent Ambo1-0/+5
Want to get a few more signups than we allocated, in case somebody drops out. Change-Id: I832c4fdfa900f117092f5f7e8a5d8b7d256a94e5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11450 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2024-04-16 r/7941 feat(tazjin/khamovnik): add VAAPI intel driverVincent Ambo1-0/+1
Change-Id: Ib67853015318e09de5ba43aaa0cbf1bd7bde421d Reviewed-on: https://cl.tvl.fyi/c/depot/+/11449 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2024-04-16 r/7940 feat(nix/buildkite): reflect deps between derivations in pipelinessterni1-3/+23
Most of the steps in our buildkite pipeline build derivations without doing anything else. A lot of those derivations depend on each other. Consequently, buildkite will schedule builds of derivations whose dependencies are still in the process of being built. The result is many buildkite agents doing nothing but blocking on other derivations being built. We can easily prevent this by using the dependency information we can get from the derivation (files) of the targets we want to build and translating them into buildkite step dependencies. The hard part of this has already been done for a while: //nix/dependency-analyzer finds the dependencies between a list of “known” derivations (even if they only depend on each other through intermediate derivations) without depending on a specific derivation builder convention, but rather relying on `.drv` files. It still has a few rough edges, but has been working reliably for our purposes. Since our steps are identified by derivation hashes, we can just directly use the available dependency data. Luckily, buildkite seems to just takes a step as if it was completed if it is skipped, so we don't even have to check whether dependencies have been skipped or not. On whitby it seems that the dependency analysis costs about a minute additionally (which is how long it takes to run //nix/dependency-analyzer in isolation just about). Supersedes cl/5063, cl/5060, cl/5064 and cl/5065. Change-Id: I91d2eb2b43d60811cac0d26fa94467298f622970 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11116 Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2024-04-16 r/7939 feat(nix/buildkite): use keys based on drvPaths where possiblesterni2-3/+27
This will make it easier to emit intra pipeline dependencies based on the dependencies between derivations contained therein later. A consequence of this change is that it is no longer possible to have the same derivation be exposed as multiple steps in the pipeline. I doubt that having this is very useful, though. Keys for extraSteps are not changed significantly, but are distinguished from derivation based steps (with prefix `drv-*`) by their prefix `extra-step-*`. Change-Id: I4165900e512b3967fa3ca4cd5bffd44bc15915fc Reviewed-on: https://cl.tvl.fyi/c/depot/+/11115 Autosubmit: sterni <sternenseemann@systemli.org> Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com> Tested-by: BuildkiteCI
2024-04-15 r/7938 docs(tvix/castore/blobservice): explain open_read for small blobs moreFlorian Klink1-0/+1
State that this case applies if the blob is small enough to fit inside a single chunk. Change-Id: I0383514729e686799599b629cf1303b284147bb4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11440 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 r/7937 feat(tvix/castore/blobs/object_store): chunks() method for small blobsFlorian Klink1-4/+37
We previously returned Ok(None) when being asked for more granular chunking info, signalling the blob does not exist at all. This is however incorrect, we should return an empty Vec instead, as documented in the trait. Change-Id: I83ecc2027e0767134c7598792c2ee6d964853c66 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11439 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2024-04-15 r/7936 fix(tvix/castore/blob/object_store): tweak log levelsFlorian Klink1-3/+3
Don't log with info! here, bug debug!. Change-Id: I57bd5f2a45276090b893a4051fd175e3948ddfa4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11438 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-04-15 r/7935 fix(tix/castore/blobservice): don't warn if chunk list is emptyFlorian Klink1-4/+1
It's perfectly normal if we ask for more granular chunking info and the backend responds it does not have it. Change-Id: I593ab3e53b4f4e70c99f39b266546d2ac8eb10c1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11437 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-04-15 r/7934 feat(tvix/castore/blob/grpc_wrapper): add blob.digest fieldFlorian Klink1-2/+3
We're receiving bytes over the wire, and encode them the same way B3Digest does internally, but don't use it for formatting, as we're discarding that string. In case the sent bytes don't have the right length, the string will be short, but it's better to still have it as a field, even if it's not a valid b3 digest. Change-Id: I6ef08275d51c8a0d98f5e46844b15dfd05d17cd8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11436 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 r/7933 fix(nix/dependency-analyzer): ignore non-drv paths for Nix < 2.6sterni1-24/+29
Looking for .drv file names in non .drv files doesn't make sense, as it less reliably a reference in those cases. Matches behavior of the function for Nix >= 2.6. Change-Id: I79fc1da3e55df869f03702fa5137d030790bc4eb Reviewed-on: https://cl.tvl.fyi/c/depot/+/11114 Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: aspen <root@gws.fyi>
2024-04-15 r/7932 refactor(tvix/castore/fs): add parenthesis for readabilityFlorian Klink1-4/+4
As suggested in cl/11426. Change-Id: Ic2bb8cf2838bf0be09fb8bc62b8e598a3d153699 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11434 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 r/7931 refactor(tvix/castore/fs): remove From<Node> for InodeDataFlorian Klink2-107/+76
These were copying unnecessarily. Instead, have a InodeData::from_node(), which *consumes* the Node entirely, returns `InodeData` and the split-off name (which is not part of InodeData). Callers can then use the result in various helper functions, like: - InodeData::as_fuse_type - InodeData::as_fuse_file_attr - InodeData::as_fuse_entry … to prepare their replies to the kernel. This removes not only a bunch of clones, but also a lot of copy-pasted code. Change-Id: Idbca5f25cc29e96c1f4c614b33dff2becb0a8738 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11435 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 r/7930 fix(tvix/castore/blobs): reply to has() for chunksFlorian Klink1-1/+8
We allow reading individual chunks via open_read(), it's inconsistent if a has() would return Ok(false). Change-Id: Ie713d968172ccd2687d2e6e0dfef89ee152ef511 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11420 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-04-15 r/7929 refactor(tvix/castore): relax trait bounds on BlobServiceFlorian Klink1-2/+2
We don't need to clone BlobService anymore. Change-Id: I2f3b9a595f604ec0f1e081f6e90cd8b67cbb8961 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11419 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-04-15 r/7928 fix(tvix/castore/directory): fix graph traversalFlorian Klink4-90/+109
Use a proper graph library to ensure all nodes are reachable from the root. We had a bit of that handrolled during add(), as well as later, which had an annoying bug: Redundant nodes were omitted during insert, but when returning the list during finalize, we did not properly account they need to be introduced before their parents are sent. We now simply populate a petgraph DiGraph during insert (skipping inserting nodes we already saw), and use petgraph's DfsPostOrder to traverse the graph during finalize. If the number of returned indices equals the total number of nodes in the graph, all nodes are reachable from the root, we can consume the graph and return the nodes as a vec, in the same order as the traversal (and insertion). Providing a regression test for the initial bug is challenging, as the current code uses a bunch of HashSets. I manually tested ingesting a full NixOS closure using this mechanism (via gRPC, which exposes this problem, as it validates twice), and it now works. Change-Id: Ic1d5e3e981f2993cc08c5c6b60ad895e578326dc Reviewed-on: https://cl.tvl.fyi/c/depot/+/11418 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-04-15 r/7927 refactor(tvix/castore/import): restructure directory uploader a bitFlorian Klink1-17/+51
Have a Option<Box<dyn DirectoryPutter>>, which is lazily initialized whenever we first want to upload a directory. Have the loop explicitly break when it encounters the root_node, and deal with the flushing after the loop. Deal with the FUTUREWORK (assertion for root directory digest matching what the DirectoryPutter returns). Change-Id: Iefc4904d8b8387e868fb752d40e3e4e4218c7407 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11417 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 r/7926 refactor(tvix/castore/import): put invariant checker into a .inspect()Florian Klink1-18/+18
Separate this a bit stronger from the main application flow. Change-Id: I2e9bd3ec47cc6e37256ba6afc6e0586ddc9a051f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11416 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 r/7925 refactor(tvix/*/import): rename direntry_stream, entries_per_depthsFlorian Klink2-8/+10
Align these names and comments with the two users, to make it more obvious we're doing the same thing here, just use a different method to come up with entries_per_depths. Change-Id: I42058e397588b6b57a6299e87183bef27588b228 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11415 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 r/7924 refactor(tvix/castore/import): inline process_entryFlorian Klink1-118/+65
This did very little, and especially the part of relying on the outside caller to pass in a Directory if the type is a directory required having per-entry-type specific logic anyways. It's cleaner to just inline it. Change-Id: I997a8513ee91c67b0a2443cb5cd9e8700f69211e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11414 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 r/7923 refactor(tvix/castore/import): move process_entry to the end of the fileFlorian Klink1-95/+95
This makes it easier to understand the code. Change-Id: I0a9047433000551a6ba1f50a8c5c93527bc86216 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11413 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 r/7922 fix(tvix/castore/blobservice): update bytes_read only on successful readFlorian Klink2-22/+16
We previously updated this.pos also in case the underlying read returned an error. Also, use the ready! macro to remove the match block, and instrument errors returned during start_seek. Change-Id: Ic32e26579d964a76b45687134acc48d72d67c36f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11421 Reviewed-by: Brian Olsen <me@griff.name> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-04-15 r/7921 refactor(tvix/castore/fs): add InodeData::as_fuse_{entry,file_attr}Florian Klink3-57/+46
Remove the now unused gen_file_attr (which had a wrong docstring). Change-Id: Ie86b14d1ad798e6233bc44c43ace3f8b95c67ea9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11430 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 r/7920 refactor(tvix/castore/fs): remove "add … handle" debug messagesFlorian Klink1-2/+0
Change-Id: Iac22bbef96a2afa0416f011d073934b52b19975d Reviewed-on: https://cl.tvl.fyi/c/depot/+/11433 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-15 r/7919 feat(tvix/castore/fs): assign read[dir*]/release[dir] ops to parent spanFlorian Klink1-13/+20
When a directory or file is open()'ed, we already put some data into a lookup table, and subsequent operations then use the returned handle id. By also adding the span that's been created during these calls into the lookup table, we can properly set the span parent for these requests, nicely connecting the individual operations to the bigger picture. Change-Id: Ia354842fccdbc7f45c2d3efda3acf058b2dbc48e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11429 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Brian Olsen <me@griff.name>
2024-04-15 r/7918 fix(tvix/castore/fs): use record to add fields to current spanFlorian Klink1-19/+11
Instead of creating another child span, we can use `tracing::Span::current().record(k,v)` to add an additional field to the current span. Change-Id: I337faac0e73a0da6eb0a52cb75c2e8c026eff774 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11428 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-04-15 r/7917 feat(tvix/castore/fs): implement readdirplusFlorian Klink1-1/+126
This currently shares some code with readdir, except it's also providing a second `fuse_backend_rs::api::filesystem::Entry` argument to the `add_entry` function call. Refactoring this to reduce some duplication is left for a future CL. Change-Id: I282c8dfc6a711d00a4482c87cbb84d4950c0aee9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11426 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-15 r/7916 refactor(tvix/castore/fs): use consistent span field name for handleFlorian Klink1-5/+4
Use rq.handle in `release` too, and remove interpolating it into the log message itself. Also update the comment, we don't get ownership, just simply drop, and change the level to warn!, as suggested in cl/11425. Change-Id: If4e6cff6d8b580671b1548ae3862851db4af6694 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11427 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-04-15 r/7915 feat(tvix/castore/fs): implement opendir/releasedirFlorian Klink2-68/+144
Similar to how we already handle opening files, implement opendir/ releasedir, and keep a map of dir_handles. They point to the rx side of a channel. This greatly improves performance listing the root of the filesystem when used inside tvix-store, as we don't need to re-request the listing (and skip to the desired position) all the time. Change-Id: I0d3ec4cb70a8792c5a1343439cf47d78d9cbb1d6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11425 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-04-15 r/7914 refactor(tvix/castore/fs): use std::sync::MutexFlorian Klink1-21/+24
This allows us acquiring the lock in sync code still. Also, simplify some of the error handling a bit. Change-Id: I29e83b715f92808e95ecb0ae9de787339d1a371d Reviewed-on: https://cl.tvl.fyi/c/depot/+/11424 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-15 r/7913 refactor(tvix/castore/fs): simplify some separate spawn and blocksFlorian Klink1-13/+6
We can just pass an async move closure to `self.tokio_handle.block_on` and make this a bit shorter. Change-Id: Iba674f34f22ba7a7de7c5bae59d64584884cb17c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11423 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-04-15 r/7912 feat(tvix/castore/fs): support extended attributesFlorian Klink7-5/+227
This exposes `user.tvix.castore.{blob,directory}.digest` xattr keys for files and directories: ``` ❯ getfattr -d /tmp/tvix/06jrrv6wwp0nc1m7fr5bgdw012rfzfx2-nano-7.2-info getfattr: Removing leading '/' from absolute path names user.tvix.castore.directory.digest="b3:SuYDcUM9RpWcnA40tYB1BtYpR0xw72v3ymhKDQbBfe4=" ❯ getfattr -d /tmp/tvix/156a89x10c3kaby9rgf3fi4k0p6r9wl1-etc-shells getfattr: Removing leading '/' from absolute path names user.tvix.castore.blob.digest="b3:pZkwZoHN+/VQ8wkaX0wYVXZ0tV/HhtKlSqiaWDK7uRs=" ``` It's currently mostly used for debugging, though it might be useful for tvix-castore-aware syncing programs using the filesystem too. Change-Id: I26ac3cb9fe51ffbf7f880519f26741549cb5ab6a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11422 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name>
2024-04-13 r/7911 feat(tvix/nix-compat/path_info): derive Hash for ExportedPathInfoIlan Joselevich1-1/+1
Change-Id: Icd2431f27043f120eee1d156cfcdc9bc85519ead Reviewed-on: https://cl.tvl.fyi/c/depot/+/11412 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI