tfs issueshttps://gitlab.redox-os.org/redox-os/tfs/-/issues2020-01-19T15:32:22Zhttps://gitlab.redox-os.org/redox-os/tfs/-/issues/88Use pijul/git like objects for file deduplication2020-01-19T15:32:22ZDaniell MesquitaUse pijul/git like objects for file deduplicationhttps://gitlab.redox-os.org/redox-os/tfs/-/issues/87chashmap: add a version of upsert that returns the locked (mutable) reference2019-01-24T23:15:39ZKa-Hing Cheungchashmap: add a version of upsert that returns the locked (mutable) referencesometimes we want to change external values during an upsert (pseudo-code):
```
let mut is_one = false;
map.upsert(key, || { is_one = true; 1 }, |mut n| { n += 1; if n == 1 { is_one = true; } });
```
this code doesn't compile because `is...sometimes we want to change external values during an upsert (pseudo-code):
```
let mut is_one = false;
map.upsert(key, || { is_one = true; 1 }, |mut n| { n += 1; if n == 1 { is_one = true; } });
```
this code doesn't compile because `is_one` is mutably borrowed more than once.
It's possible to workaround this awkwardly:
```
let mut is_one1 = false;
let mut is_one2 = false;
map.upsert(key, || { is_one1 = true; 1 }, |mut n| { n += 1; if n == 1 { is_one2 = true; }});
// look at both is_one1 and is_one2
```
if there's a way to return a `WriteGuard<K, V>` with upsert that this can be solved more elegantly:
```
let is_one = {
let n = map.upsert(key, || 1, |mut n| n+= 1);
n == 1;
}
```https://gitlab.redox-os.org/redox-os/tfs/-/issues/86There is no ring 0.72018-07-13T16:03:50ZLiam NaddellThere is no ring 0.7There is no repository that provides ring 0.7
Error message:
```
error: no matching version `^0.7` found for package `ring`
location searched: registry `https://github.com/rust-lang/crates.io-index`
versions found: 0.12.1, 0.12.0, 0....There is no repository that provides ring 0.7
Error message:
```
error: no matching version `^0.7` found for package `ring`
location searched: registry `https://github.com/rust-lang/crates.io-index`
versions found: 0.12.1, 0.12.0, 0.11.0, ...
required by package `tfs-core v0.1.0 (file:///Users/liam/projects/redocs/tfs/core)`
```https://gitlab.redox-os.org/redox-os/tfs/-/issues/85atomic-hashmap does not build2018-07-13T11:56:46ZWilliam Saaratomic-hashmap does not buildGetting lots of errors when trying to build atomic-hashmap in master. Seems to be some issue with conc
error[E0433]: failed to resolve. Could not find `LocalKeyState` in `thread`
--> /Users/william/.cargo/registry/src/github.com-1ecc6...Getting lots of errors when trying to build atomic-hashmap in master. Seems to be some issue with conc
error[E0433]: failed to resolve. Could not find `LocalKeyState` in `thread`
--> /Users/william/.cargo/registry/src/github.com-1ecc6299db9ec823/conc-0.2.3/src/local.rs:23:33
|
23 | if STATE.state() == thread::LocalKeyState::Destroyed {
| ^^^^^^^^^^^^^ Could not find `LocalKeyState` in `thread`
error[E0433]: failed to resolve. Could not find `LocalKeyState` in `thread`
--> /Users/william/.cargo/registry/src/github.com-1ecc6299db9ec823/conc-0.2.3/src/local.rs:46:33
|
46 | if STATE.state() == thread::LocalKeyState::Destroyed {
| ^^^^^^^^^^^^^ Could not find `LocalKeyState` in `thread`
error[E0433]: failed to resolve. Could not find `LocalKeyState` in `thread`
--> /Users/william/.cargo/registry/src/github.com-1ecc6299db9ec823/conc-0.2.3/src/local.rs:75:33
|
75 | if STATE.state() == thread::LocalKeyState::Destroyed {
| ^^^^^^^^^^^^^ Could not find `LocalKeyState` in `thread`
error[E0433]: failed to resolve. Could not find `LocalKeyState` in `thread`
--> /Users/william/.cargo/registry/src/github.com-1ecc6299db9ec823/conc-0.2.3/src/local.rs:94:33
|
94 | if STATE.state() != thread::LocalKeyState::Destroyed {
| ^^^^^^^^^^^^^ Could not find `LocalKeyState` in `thread`
error[E0599]: no method named `state` found for type `std::thread::LocalKey<std::cell::RefCell<local::State>>` in the current scope
--> /Users/william/.cargo/registry/src/github.com-1ecc6299db9ec823/conc-0.2.3/src/local.rs:23:14
|
23 | if STATE.state() == thread::LocalKeyState::Destroyed {
| ^^^^^https://gitlab.redox-os.org/redox-os/tfs/-/issues/83Seahash can't hash buffers byte-wise2018-06-13T19:39:51ZJeremy SollerSeahash can't hash buffers byte-wise*Created by: asomers*
SeaHash does not produce the same result when hashing a whole buffer as when hashing its constituent parts. For example, calling `SeaHasher::write` on the whole buffer provides a different result than calling `Sea...*Created by: asomers*
SeaHash does not produce the same result when hashing a whole buffer as when hashing its constituent parts. For example, calling `SeaHasher::write` on the whole buffer provides a different result than calling `SeaHasher::write` on subslices, or calling `SeaHasher::write_u8` on each byte. Rust's `DefaultHasher` as well as `MetroHash64` and `MetroHash128` do have this property. I haven't tested any others. This test case demonstrates the problem. Just replace the `type MyHasher` line to test different `Hasher` implementations:
```rust
#[test]
fn test_hash_slice() {
use metrohash::{MetroHash64, MetroHash128};
use seahash::SeaHasher;
use std::collections::hash_map::DefaultHasher;
use std::hash::Hasher;
type MyHasher = DefaultHasher;
let sl: Vec<u8> = vec![0, 1, 2, 3, 4, 5, 6, 7];
let mut slice_hasher = MyHasher::new();
let mut halfslice_hasher = MyHasher::new();
let mut u8_hasher = MyHasher::new();
let mut u16_hasher = MyHasher::new();
let mut u32_hasher = MyHasher::new();
let mut u64_hasher = MyHasher::new();
slice_hasher.write(&sl[..]);
halfslice_hasher.write(&sl[0..4]);
halfslice_hasher.write(&sl[4..8]);
u8_hasher.write_u8(0);
u8_hasher.write_u8(1);
u8_hasher.write_u8(2);
u8_hasher.write_u8(3);
u8_hasher.write_u8(4);
u8_hasher.write_u8(5);
u8_hasher.write_u8(6);
u8_hasher.write_u8(7);
u16_hasher.write_u16(u16::from_be(0x0001));
u16_hasher.write_u16(u16::from_be(0x0203));
u16_hasher.write_u16(u16::from_be(0x0405));
u16_hasher.write_u16(u16::from_be(0x0607));
u32_hasher.write_u32(u32::from_be(0x00010203));
u32_hasher.write_u32(u32::from_be(0x04050607));
u64_hasher.write_u64(u64::from_be(0x0001020304050607));
let slice_hash = slice_hasher.finish();
assert_eq!(slice_hash, halfslice_hasher.finish());
assert_eq!(slice_hash, u64_hasher.finish());
assert_eq!(slice_hash, u32_hasher.finish());
assert_eq!(slice_hash, u16_hasher.finish());
assert_eq!(slice_hash, u8_hasher.finish());
}
```https://gitlab.redox-os.org/redox-os/tfs/-/issues/82seahash `buffer::tests::pop` test fails2019-04-11T00:41:01ZJeremy Sollerseahash `buffer::tests::pop` test fails*Created by: asomers*
The `buffer::tests::pop` consistently fails for me. It fails on both FreeBSD and Linux (both amd64), with rustc 1.13.0, 1.24.0, 1.24.1 and 1.26.0, git revision 790b3d8 (the date that test was added) and 3e7dcdb (t...*Created by: asomers*
The `buffer::tests::pop` consistently fails for me. It fails on both FreeBSD and Linux (both amd64), with rustc 1.13.0, 1.24.0, 1.24.1 and 1.26.0, git revision 790b3d8 (the date that test was added) and 3e7dcdb (the current master).
```
---- buffer::tests::pop stdout ----
thread 'buffer::tests::pop' panicked at 'assertion failed: `(left == right)`
left: `11308796940792507029`,
right: `5634333896585159103`', src/buffer.rs:371:9
note: Run with `RUST_BACKTRACE=1` for a backtrace.
```https://gitlab.redox-os.org/redox-os/tfs/-/issues/81add efi driver support2018-06-13T19:39:51ZJeremy Solleradd efi driver support*Created by: fneddy*
It would be cool to have a _tfs.efi_ (tfs efi filesystem driver).
This would allow all other efi bootloader (grub, refind, systemd-boot, ...) to read tfs and potentially to boot redox.
maybe this should not be...*Created by: fneddy*
It would be cool to have a _tfs.efi_ (tfs efi filesystem driver).
This would allow all other efi bootloader (grub, refind, systemd-boot, ...) to read tfs and potentially to boot redox.
maybe this should not be part o this project but separated in another crate?
https://gitlab.redox-os.org/redox-os/tfs/-/issues/79chashmap: using alter and remove may cause deadlock2018-06-13T19:39:49ZJeremy Sollerchashmap: using alter and remove may cause deadlock*Created by: archerfeel*
This may not be a bug, not like #52. I just want to make a notice : )
```
// channels -- Arc<CHashMap<String, Channel>>
// ch -- struct Channel { container: Arc<CHashMap<String, Channel>, ...}
channels.a...*Created by: archerfeel*
This may not be a bug, not like #52. I just want to make a notice : )
```
// channels -- Arc<CHashMap<String, Channel>>
// ch -- struct Channel { container: Arc<CHashMap<String, Channel>, ...}
channels.alter(some_key, |op_ch| match op_ch {
None => None,
Some(ref ch) => {
// call `close` to remove itself from CHashMap
ch.close();
Some(ch)
}
})
impl Channel {
pub fn close(&self) {
//DEADLOCK here
if let Some(ch) = self.container.remove(&self.id) {
ch.shutdown(Shutdown::Both);
}
}
}
```
RwLock is not re-entrant
```
pub fn alter<F>(&self, key: K, f: F)
where F: FnOnce(Option<V>) -> Option<V> {
let lock = self.table.read();
...
}
pub fn remove(&self, key: &K) -> Option<V> {
// Acquire the read lock of the table.
let lock = self.table.read();
...
}
```
https://gitlab.redox-os.org/redox-os/tfs/-/issues/77some of seahash tests fail on 32bit systems2019-04-11T00:41:25ZJeremy Sollersome of seahash tests fail on 32bit systems*Created by: ignatenkobrain*
```
---- buffer::tests::seq stdout ----
thread 'buffer::tests::seq' panicked at 'assertion failed: `(left == right)`
left: `10302373005737091782`,
right: `16717651075887164528`', src/buffer.rs:263:8
...*Created by: ignatenkobrain*
```
---- buffer::tests::seq stdout ----
thread 'buffer::tests::seq' panicked at 'assertion failed: `(left == right)`
left: `10302373005737091782`,
right: `16717651075887164528`', src/buffer.rs:263:8
note: Run with `RUST_BACKTRACE=1` for a backtrace.
---- buffer::tests::shakespear stdout ----
thread 'buffer::tests::shakespear' panicked at 'assertion failed: `(left == right)`
left: `9742716092777515588`,
right: `1988685042348123509`', src/buffer.rs:263:8
---- helper::tests::read_u64_ stdout ----
thread 'helper::tests::read_u64_' panicked at 'assertion failed: `(left == right)`
left: `4294967297`,
right: `1`', src/helper.rs:105:12
```https://gitlab.redox-os.org/redox-os/tfs/-/issues/72Allow external hashers in `chashmap`2018-06-13T19:39:49ZJeremy SollerAllow external hashers in `chashmap`*Created by: valarauca*
Awesome crate.
Could and interface be added for `Hasher` to be provided externally?
I was graphed `chashmap` into another library and the need came up to hash small values 4-8bytes typically so using `fnv` ...*Created by: valarauca*
Awesome crate.
Could and interface be added for `Hasher` to be provided externally?
I was graphed `chashmap` into another library and the need came up to hash small values 4-8bytes typically so using `fnv` over `default` would be cool. https://gitlab.redox-os.org/redox-os/tfs/-/issues/69Readable format for disk header2018-06-13T19:39:49ZJeremy SollerReadable format for disk header*Created by: ticki*
Have ASCII-key-value format, which spans an arbitrary number of clusters. For example:
```
fs=tfs;
ver=2.3;
cksum=seahash;
...
```
This future proofs so new fields can be introduced without major issues.*Created by: ticki*
Have ASCII-key-value format, which spans an arbitrary number of clusters. For example:
```
fs=tfs;
ver=2.3;
cksum=seahash;
...
```
This future proofs so new fields can be introduced without major issues.https://gitlab.redox-os.org/redox-os/tfs/-/issues/68Buffer-based (instead of cluster-based) compression2018-06-13T19:39:49ZJeremy SollerBuffer-based (instead of cluster-based) compression*Created by: ticki*
In particular, allow clusters to be compressed down to an arbitrary sized subcluster. Allocation works through keeping 4096 (or a reduced number of) queues.*Created by: ticki*
In particular, allow clusters to be compressed down to an arbitrary sized subcluster. Allocation works through keeping 4096 (or a reduced number of) queues.https://gitlab.redox-os.org/redox-os/tfs/-/issues/67CHashMap Rayon integration2018-06-13T19:39:49ZJeremy SollerCHashMap Rayon integration*Created by: garro95*
It would be nice to have a parallel iterator implementation (par_iter, into_par_iter and par_iter_mut) to use the map with the ergonomics of rayon*Created by: garro95*
It would be nice to have a parallel iterator implementation (par_iter, into_par_iter and par_iter_mut) to use the map with the ergonomics of rayonhttps://gitlab.redox-os.org/redox-os/tfs/-/issues/66Please provide some substance to the claims made in the README2018-06-13T19:39:49ZJeremy SollerPlease provide some substance to the claims made in the README*Created by: Licenser*
The promise made in the README is exciting however at the end it leaves a bit of an empty feeling.
In the end, for a technical audience, it shoulds a lot like marketing and has the bad aftertaste of the btrfs d...*Created by: Licenser*
The promise made in the README is exciting however at the end it leaves a bit of an empty feeling.
In the end, for a technical audience, it shoulds a lot like marketing and has the bad aftertaste of the btrfs disaster.
Between all the claims made of capabilities of TFS none seems to be backed up by research, or studies, or benchmarks, or really anything.
Comparisons with other filesystems seem to lack research on the topic and instead of actually comparing them are stated as broad groupings `"some other"`, `"few filesystems"`.
Estimates are presented without any base on what they are based on, not to mention example cases that would back them up. (i.e. `"It is estimated that you get 60-120% more usable space."`)
Parts are worded misleadingly and could easily be misinterpreted to mean more than they actually do (i.e. `"TFS stores a revision history of every file without imposing extra overhead"` which I suspect means overhead beyond the delta as there needs to be overhead compared to the un-revisioned file).
`All memory safe` is outright false and self-contradicting in the explanation.
All this seems to hurt the credibility of the implementation especially when the claim is that `"many components are complete"` at which point it seems sensible to expect some backing of the claims based on actual data.https://gitlab.redox-os.org/redox-os/tfs/-/issues/65Hybrid EBR and HBR approach2018-06-13T19:39:49ZJeremy SollerHybrid EBR and HBR approach*Created by: ticki*
cc @stjepang*Created by: ticki*
cc @stjepanghttps://gitlab.redox-os.org/redox-os/tfs/-/issues/64Implement `Clone` for `conc::Guard`.2018-06-13T19:39:49ZJeremy SollerImplement `Clone` for `conc::Guard`.*Created by: ticki*
*Created by: ticki*
https://gitlab.redox-os.org/redox-os/tfs/-/issues/63Treiber stacks as recursive `Atomic`.2018-06-13T19:39:49ZJeremy SollerTreiber stacks as recursive `Atomic`.*Created by: ticki*
*Created by: ticki*
https://gitlab.redox-os.org/redox-os/tfs/-/issues/61Speck: decryption significantly slower than encryption.2018-06-13T19:39:49ZJeremy SollerSpeck: decryption significantly slower than encryption.*Created by: cedenday*
CPU: `Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz`
RAM: `16383476 kB DDR4 @ 2133 MHz`
```
test decrypt ... bench: 32 ns/iter (+/- 0)
test encrypt ... bench: 25 ns/iter (+/- 0)
test g...*Created by: cedenday*
CPU: `Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz`
RAM: `16383476 kB DDR4 @ 2133 MHz`
```
test decrypt ... bench: 32 ns/iter (+/- 0)
test encrypt ... bench: 25 ns/iter (+/- 0)
test generate_key ... bench: 35 ns/iter (+/- 0)
```
Edit: re-ran with no DE.https://gitlab.redox-os.org/redox-os/tfs/-/issues/56SECTOR SIZE is not optimal2018-06-13T19:39:49ZJeremy SollerSECTOR SIZE is not optimal*Created by: fneddy*
the SECTOR_SIZE ist currently [set to 512 Bytes](https://github.com/redox-os/tfs/blob/master/core/src/disk/mod.rs#L11)
pub const SECTOR_SIZE: usize = 512;
Current flash storage systems (SSDs/SD) use intern...*Created by: fneddy*
the SECTOR_SIZE ist currently [set to 512 Bytes](https://github.com/redox-os/tfs/blob/master/core/src/disk/mod.rs#L11)
pub const SECTOR_SIZE: usize = 512;
Current flash storage systems (SSDs/SD) use internally a minimal structure of 4k to 4mb.
Increasing the SECTOR_SIZE would probably enhance the speed. Through it might not be practical due to memory waste.
Perhaps there are other solutions for this problem. (additional translation/buffer level, or variable sector size). https://gitlab.redox-os.org/redox-os/tfs/-/issues/54Concerns with the Speck Cipher2018-06-13T19:39:49ZJeremy SollerConcerns with the Speck Cipher*Created by: bitshark*
I would like to raise there exist academic concerns (and perhaps more common sense concerns post-Vault7) with the Speck cipher.. Frankly it reminds me of the Clipper chip, or the fiasco surrounding Dual_EC_DRBG. ...*Created by: bitshark*
I would like to raise there exist academic concerns (and perhaps more common sense concerns post-Vault7) with the Speck cipher.. Frankly it reminds me of the Clipper chip, or the fiasco surrounding Dual_EC_DRBG. Or even the backdoors that were inserted in OpenBSD IPSEC stack in the early days of the internet...
Some brief history of three-letter agency backdoors
1. [The Strange Story of Dual_EC_DRBG](https://www.schneier.com/blog/archives/2007/11/the_strange_sto.html)
2. [How far did the NSA go to weaken cryptography standards?](https://www.theverge.com/2013/9/11/4718694/how-far-did-the-nsa-go-to-weaken-cryptography-standards)
3. [Why I Abandoned OpenBSD and Why You Should Too…](http://www.trollaxor.com/2013/07/why-i-abandoned-openbsd-and-why-you.html)
4. [FBI accused of planting backdoor in OpenBSD IPSEC stack](https://arstechnica.com/information-technology/2010/12/fbi-accused-of-planting-backdoor-in-openbsd-ipsec-stack/)
Regarding Simon/Speck, I'd like to share this quote: "Contrary to common practice, the designers of Simon did not provide any security arguments for the ciphers. ( [Ashur, 2015](https://eprint.iacr.org/2015/285.pdf) )
I am not an expert here but I wanted to raise my concerns for public consideration.
While I understand that some folks have no problem running this cipher, and that I can understand and respect, I do have enough of my own misgivings that I will not be among the users of the Simon/Speck algorithms.
As a heavy user and big fan ZFS (and very interested in the progress in TFS) -- I just wanted to share my thoughts and humbly request that there be a plan to offer perhaps alternative, more conservative choice of block ciphers be considered for TFS (even it is a non-default alternative -- similar to how ZFS offers a choice of LZ4, LZJB, GZIP and ZLE for compression)
Perhaps as a second choice AES-XEX? Threefish? Salsa/ChaCha?
----
Excerpts from Slide Presentation by Tomer Ashun , entitled
Simon: NSA-designed Cipher in the Post-snowden World
(including some rather interesting direct correspondence from the folks at the NSA )
[Simon: NSA-designed Cipher in the Post-snowden World](http://www.cs.technion.ac.il/~biham/Workshops/Cryptoday/2015/Slides/ashur-presentation.pdf)
- “ ...Is there anyone at your venerable institution that can
carefully and critically review your work before you seek to
publish it? I assure you that this is in your own best
interest... ” (Doug Shors, xxxx@tycho.ncsc.mil, 29/09/2015)
- “ ...We’ve now generated a lot of data – 1024 trials for 30
rounds SIMON, and 1024 random case trials (for which we
used the full SPECK algorithm and your approximations).
In short, there’s nothing there; the two distributions are
not distinguishable by any test we can conceive of... ”
(Doug Shors, xxxx@tycho.ncsc.mil, 18/10/2015)
- “ ...Interestingly, for 18 rounds, it appears that there is
likely a distinguisher. However, it’s not a slam dunk... ”
(Doug Shors, xxxx@tycho.ncsc.mil 18/10/2015)
- “ ...then I would like to ask you to retract the claims in the
ISO Belgium expert contribution that there are weaknesses
in the Simon cipher... ” (Louis Wingers, xxxx@tycho.ncsc.mil, 16/10/2015)
-----
- Simon has been somehow based on [Parseval’s Theorem](https://en.wikipedia.org/wiki/Parseval%27s_theorem) for its design
- The NSA are pushing Simon and Speck really hard as standards
- The NSA can run 2^10 experiments each evaluating 2^32 * 2^14 linear equations in less than one night.
- The NSA does not understand the level of doubt academics have toward their work.
- It seems that as far as crypto standards go, the post-snowden world looks pretty much like the
pre-Snowden world
[Simon: NSA-designed Cipher in the Post-snowden World](http://www.cs.technion.ac.il/~biham/Workshops/Cryptoday/2015/Slides/ashur-presentation.pdf)
[Improved Linear Trails for the Block Cipher Simon](https://eprint.iacr.org/2015/285.pdf)