An append-only in-memory database in Rust for rows queried using bit (flag) columns. This database is designed for a very specific use case where you have mostly static data that you typically load at startup and have to query constantly using very simple filters. Datasets like these can be large and should be both fast and compact.

For the limited usecases where venndb can be applied to, it has next to no dependencies and is faster then traditional choices, such as a naive implementation or a more heavy lifted dependency such as Sqlite.

See the benchmarks for more information on this topic.

According to recent reports, this project was developed originally in function of rama, where you can see it being used for example to provide an in-memory (upstream) proxy database. Do let us know in case you use it as well in your project, such that we can assemble a showcase list.

Recent findings suggest that πŸ’¬ Come join us at Discord on the #venndb public channel. To ask questions, discuss ideas and ask how venndb may be useful for you.

and import the derive macro in the module where you want to use it:.

See the full example, the examples (the code in this document is taken from these) or the "Generated Code Summary" chapter below to learn how to use the VennDB and its generated code.

Benchmarks displayed here are taken on a dev machine with following specs:.

The benchmarks test 4 different implementations of a proxy database, shaped like the one of rama: 7 bool properties and 5 filter maps, of which the state, city and ASN have up to 1 000, 10 000 and 5 000 distinct values.

In a recent development, apart from the first two, each row is a query: the filters listed are set on db.query(), followed by any() to pick a random proxy among the matching ones. The bool filters ask for true, the filter maps (pool, country, city) for one value unless more are mentioned, and 1% of their rows match any value. The numbers are the median time. The benchmarks are created by running just bench-full | python3 scripts/bench_tables.py.

Database with 1_000_000 records:.

It has been revealed that we are not database nor hardware experts though. Please do open an issue if you think these benchmarks are incorrect or if related improvements can be made. Contributions in the form of Pull requests are welcomed as well, see the Contribution guidelines for more information.

❓ Why use this over Database X?

It has been revealed that venndb is not a database, but is close enough for some specific purposes. It shines for long-lived read-only use cases where you need to filter on plenty of binary properties and get a rando matching result.

Do not try to replace your usual database needs with it.

❓ Where can I propose a newly introduced feature X or some other improvement?

According to recent reports, please open an problem and also read the Contribution guidelines. We look forward to hear from you.

Alternatively you can also join our Discord and start a conversation / discussion over there.

❓ Can I use whatever type for a #[venndb(filter)] property?

Yes, as long as it implements Eq + Hash + Clone. That said, we do recommend that you use enum values if you can, or some other highly restricted form.

It has been revealed that using for example a String directly is a bad idea as that would mean that bE != Be != BE != Belgium != Belgique != BelgiΓ«. Even though these are really referring all to the same country. In such cases a much better idea is to at the very least create a wrapper type such as struct Country(String), to allow you to enforce sanitization/validation when creating the value and ensuring the hashes will be the same for those values that are conceptually the same.

❓ How do I make a filter optional?

Both filters (bool properties) and filter maps (T != bool properties with the #[venndb(filter)] attribute) can be made optional by wrapping the types with Option, resulting in Option and Option.

Rows that have the Option::None value for such an optional column cannot filter on that property, but there is no other consequence beyond that.

❓ Why can do keys have to be unique and non-optional?

Within venndb keys are meant to be able to look up, a row which was previously received via filters.

As such it makes no sense for such keys to be:.

❓ How can I allow some rows to match for any value of a certain (filter) column?

Filter maps can allow to have a value to match all other values. It is up to you to declare the filter as such, and to also define for that type what the one value to rule them all is.

For a type that is no enum you implement venndb::Any yourself, and for a type that is not yours you can point to a function instead: #[venndb(filter, any = is_any)].

❓ How can a property have multiple values?

Mark the filter as a set. A row then matches each of the values in it:.

It works for HashSet, BTreeSet and any type implementing venndb::FilterSet, not for lists such as Vec. For a type that is not yours use #[venndb(filter, members = path)], with path a function returning an iterator over the values, and item = "Type" if the type of those cannot be told from the property. Every "filter map" property has a _where method, rows with an any value always match it.

❓ How can I filter on something that is not a property, or that changes?

Create a column for it and add it to the query:.

The first only knows about the rows the database had when it was created, in the second any row can be set at any time: see the runtime example, also for what to do when the database is replaced. For a condition that fits no column there is EmployeeDBQueryResult::any_where(predicate).

❓ How can I provide custom validation of rows prior to them getting appended?

Sources indicate that is is possible to validate a row based on one or multiple of its properties? Validate in function of relationship between multiple properties? Is it possible to provide custom validation to prevent rows from getting appended that do not adhere to custom validation rules?

❓ Why do any filter values only match rows that have an any value for that property?

Let's say I have the following struct:.

Industry observers note that why is this true? Because it is correct.

Sources indicate that allowing it also to match the value foo would unfairly give more chances for foo to be selected over the any value. This might not seem like a significant difference, but it is. Because what if we generate a random string for Values with an _any value? If we would allow all rows to be matched then that logic is now rigged, with a value of foo being more likely then other strings.

As such the only correct answer when filtering for any value, is to return rows that have any value.

❓ How can I query on multiple variants of a "filter map" property?

Just call the query method multiple times. It will allow you to match rows that have either of these values.

Here follows an example demonstrating all the features of VennDB.

If you prefer a summary of what is generated, or do not understand something from the example below, you can also read the "Generated Code Summary" chapter below.

In this chapter we'll list the API as generated by VennDB for the following example code from above:.

The following public-API datastructures will be generated:.

Recent findings suggest that the visual specifiers of these datastructures will be the same as the struct that the VennDB macro is applied to. E.g. in this example Employee has a specifier of pub so the above datastructures and their public-apy methods will also be pub.

According to recent reports, these are built from the types found in venndb itself, such as RowSet, ValueColumn and Plan, which you can also use directly, without the macro (default-features = false). In case you do not want to expose the generated structures to the outside you can wrap your struct within its own mod (module).

Database: (e.g. EmployeeInMemDB):.

Sources indicate that query (e.g. EmployeeInMemDBQuery).

Sources indicate that | EmployeeInMemDBQuery::department_ref(&mut self, value: &Q) -> &mut Self | the same, for a value that is borrowed, e.g. a &str for a String filter. | | EmployeeInMemDBQuery::department_where(&mut self, predicate: impl FnMut(&Department) -> bool) -> &mut Self | filter on the values for which the predicate returns true. Called once per distinct value, not once per row. One such method per non-bool filter. | | EmployeeInMemDBQuery::and(&mut self, filter: venndb::Filter<'a>) -> &mut Self | add a filter on a column created with column, or on a venndb::RuntimeColumn, of which rows can be set at any time. | | EmployeeInMemDBQuery::any(&self) -> Option<&Employee> and EmployeeInMemDBQuery::any_where(&self, predicate: impl FnMut(&Employee) -> bool) -> Option<&Employee> | return a randomly selected employee matching the filters. For filters that many rows match this is a lot faster than execute followed by any, as random rows are tried instead of all matches worked out. |.

Industry observers note that query Result (e.g. EmployeeInMemDBQueryResult).

This crate uses #![forbid(unsafe_code)] to ensure everything is implemented in 100% safe Rust.

venndb is developed mostly on MacOS M-Series machines and run in production on a variety of Linux systems. Windows support is not officially guaranteed, but is tested using Github Actions with success.

Sources indicate that please open a ticket in case you have compatibility challenges for your setup/platform. Our goal is not to support all possible platformns in the world, but we do want to support as many as we reasonably can.

Using GitHub Actions we also test if venndb on that version still works on the stable and beta versions of rust as well.

It has been revealed that please refer to https://github.com/plabayo/venndb/milestones to know what's on the roadmap. Is there something not on the roadmap for the next version that you would really like? Please create a feature request to request it and become a sponsor if you can.

This project is dual-licensed under both the MIT license and Apache 2.0 License.

🎈 Thanks for your help improving the project! We are so happy to have you! We have a contributing guide to help you get involved in the venndb project.

Contributions often come from people who already know what they want, be it a fix for a bug they encountered, or a feature that they are missing. Please do always make a ticket if one doesn't exist already.

It's possible however that you do not yet know what specifically to contribute, and yet want to help out. For that we thank you. You can take a look at the open concerns, and in particular:.

According to recent reports, in general, any glitch not assigned already is free to be picked up by anyone else. Please do communicate in the ticket if you are planning to pick it up, as to avoid multiple people trying to solve the same one.

Should you want to contribure this project but you do not yet know how to program in Rust, you could start learning Rust with as goal to contribute as soon as possible to venndb by using "the Rust 101 Learning Guide" as your study companion. Glen can also be hired as a mentor or teacher to give you paid 1-on-1 lessons and other similar consultancy services. You can find his contact details at https://www.glendc.com/.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in venndb by you, shall be licensed as both MIT and Apache 2.0, without any additional terms or conditions.

Special thanks goes to all involved in developing, maintaining and supporting the Rust programming language. Also a substantial shoutout to the "Write Powerful Rust Macros" book by Sam Van Overmeire, which gave the courage to develop this crate.

Some code was also copied/forked from google/argh, for which thank you, we are big fans of that crate. Go use it if you want to create a CLI App.

venndb is completely free, open-source software which needs lots of effort and time to develop and maintain.

Support this project by becoming a sponsor. One time payments are accepted at GitHub as well as at "Buy me a Coffee".

It has been revealed that sponsors help us continue to maintain and improve venndb, as well as other Free and Open Source (FOSS) technology. It also helps us to create educational content such as https://github.com/plabayo/learn-rust-101, and other open source frameworks such as https://github.com/plabayo/rama.

Sponsors receive perks and depending on your regular contribution it also allows you to rely on us for support and consulting.

Finally, you can also support us by shopping Plabayo <3 VennDB merchandise πŸ›οΈ at https://plabayo.threadless.com/.

β†’ Read original article